🛍️ ShopStack Microservices Project

A comprehensive e-commerce microservices architecture built with containerized services, featuring modern CI/CD pipelines, monitoring, and observability.

📋 Table of Contents

🏗️ Architecture Overview
🚀 Services
🔧 Development Setup
🐳 Docker & Local Development
🚀 CI/CD Pipeline
☸️ Kubernetes Deployment
📊 Monitoring & Observability
🔒 Security
📁 Project Structure

🏗️ Architecture Overview

ShopStack implements a microservices-based e-commerce platform with the following core components:

🎯 Business Services

Web Frontend - Nginx-based UI gateway with reverse proxy
User Service - User authentication and management (MongoDB + Redis)
Catalog Service - Product catalog management (MongoDB)
Cart Service - Shopping cart functionality (Redis)
Payment Service - Payment processing with RabbitMQ messaging
Shipping Service - Order fulfillment and shipping (MySQL)
Ratings Service - Product rating and review system (MySQL)
Dispatch Service - Order processing and dispatch via RabbitMQ

🛠️ Infrastructure Services

MongoDB - Primary database for User, Catalog, and Cart services
Redis - Caching layer for User and Cart services
MySQL - Database for Shipping and Ratings services
RabbitMQ - Message queuing for Payment and Dispatch services

📊 Observability Stack

OpenTelemetry - Distributed tracing and metrics
Elasticsearch - Centralized logging and search
Kibana - Log visualization and analysis
Prometheus - Metrics collection and alerting
Grafana - Metrics visualization (disabled in favor of Kibana)
Fluentd/Fluentbit - Log aggregation and forwarding

🚀 CI/CD & Deployment

AWS CodeBuild - Automated build and deployment
Amazon ECR - Container registry
ArgoCD - GitOps continuous deployment
Slack Integration - Build and deployment notifications

🚀 Services

🌐 Web Frontend

Technology: Nginx + React SPA
Port: 8080 (exposed)
Features: Reverse proxy, static asset serving, service routing
Health Check: / endpoint
Dependencies: catalogue, user, shipping, payment
Monitoring: Instana APM integration

👤 User Service

Technology: Node.js/Express
Database: MongoDB + Redis
Features: JWT authentication, user management, profile management
Health Check: /health endpoint
Dependencies: mongodb, redis

📚 Catalog Service

Technology: Node.js/Express
Database: MongoDB
Features: Product catalog, search, inventory management
Health Check: /health endpoint
Dependencies: mongodb

🛒 Cart Service

Technology: Node.js/Express
Database: Redis (session storage)
Features: Shopping cart, session management, cart persistence
Health Check: /health endpoint
Dependencies: redis

💳 Payment Service

Technology: Node.js/Express
Message Queue: RabbitMQ
Features: Payment processing, multiple gateway support (WorldPay)
Health Check: /health endpoint
Dependencies: rabbitmq
Security: PCI compliance considerations

📦 Shipping Service

Technology: Node.js/Express
Database: MySQL
Features: Order fulfillment, shipping calculation, tracking
Health Check: /health endpoint
Dependencies: mysql

⭐ Ratings Service

Technology: Node.js/Express
Database: MySQL
Features: Product ratings, reviews, averaging system
Health Check: /_health endpoint (different path)
Environment: Production mode enabled
Dependencies: mysql

🚚 Dispatch Service

Technology: Node.js/Express
Message Queue: RabbitMQ
Features: Order processing, event dispatch, workflow orchestration
Dependencies: rabbitmq
Note: No health check endpoint

🔧 Development Setup

Prerequisites

Docker & Docker Compose
Node.js 16+ (for local development)
AWS CLI (for CI/CD)
Git (for version control)

Local Development

# Clone the repository
git clone https://github.com/Uj5Ghare/Kops-Microservices-Project.git
cd Kops-Microservices-Project

# Copy environment file
cp .env.example .env

# Edit environment variables
nano .env

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f [service-name]

# Stop services
docker-compose down

Environment Variables

# Required for Docker Compose
REPO=shopstack                    # Docker repository name
TAG=1.0.0                        # Docker image tag
INSTANA_AGENT_KEY=your-key       # Instana monitoring key

# Required for CI/CD
AWS_ACCOUNT_ID=your-aws-account-id
PROJECT=shopstack
TARGET_BRANCH=k8s/staging  # or k8s/preprod

🐳 Docker & Local Development

Docker Compose Services

MongoDB: Custom image built from mongo/ directory
Redis: Official Redis 6.2-alpine image
RabbitMQ: Official RabbitMQ 3.8-management-alpine image
MySQL: Custom image built from mysql/ directory with NET_ADMIN capabilities
Microservices: Built from individual service directories with standardized health checks
Network: Shared shopstack network for inter-service communication

Health Checks

All services implement standardized health checks using:

curl -H "X-INSTANA-SYNTHETIC: 1" -f http://localhost:8080/health

Exceptions:

Web Frontend: Uses / endpoint instead of /health
Ratings Service: Uses /_health endpoint
Dispatch Service: No health check endpoint

Service Dependencies

Web → Catalogue, User, Shipping, Payment
User → MongoDB, Redis
Catalogue → MongoDB
Cart → Redis
Payment → RabbitMQ
Shipping → MySQL
Ratings → MySQL
Dispatch → Rabbitmq

🚀 CI/CD Pipeline

AWS CodeBuild Configuration

Build Spec: buildspec.yml
Triggers: Git push to k8s/staging and k8s/preprod
Environments: Staging and Production
Registry: Amazon ECR

Build Process

Pre-build: ECR login and change detection
Build: Docker image creation with multi-platform support
Post-build: Git operations and Slack notifications
Deploy: Helm chart updates and ArgoCD sync

Automated Versioning

Semantic Versioning: Automatic patch version increment
Tag Strategy: MAJOR.MINOR.PATCH format
Rollback Support: Previous version tracking

Slack Integration

Build Notifications: Start and completion alerts
Deployment Notifications: Environment-specific channels
Failure Alerts: Immediate error reporting

☸️ Kubernetes Deployment

🌳 Branching Strategy & GitOps Workflow

The ShopStack project uses a GitOps-based branching strategy with environment-specific branches:

📁 Repository Structure:
├── k8s/staging     → Staging environment deployments
├── k8s/preprod     → Production environment deployments  
└── main/develop    → Development (feature branches)

🔄 Deployment Architecture

Git Repository Structure

Kops-Microservices-Project/
├── 📁 argocd/           # ArgoCD Application manifests
│   ├── cart.yaml        # Cart service ArgoCD app
│   ├── catalogue.yaml   # Catalog service ArgoCD app
│   ├── user.yaml        # User service ArgoCD app
│   └── ...              # Other services
├── 📁 helm/             # Helm charts for each service
│   ├── cart/            # Cart service Helm chart
│   │   ├── Chart.yaml   # Chart metadata
│   │   ├── values.yaml  # Default configuration
│   │   └── templates/   # Kubernetes templates
│   ├── catalogue/       # Other service charts
│   └── ...
├── 📁 valuefiles/       # Environment-specific Helm values
│   ├── argocd-values.yaml    # ArgoCD configuration
│   ├── prometheus-values.yaml # Monitoring config
│   └── ...                  # Other infrastructure configs
└── 📁 [service-dirs]/  # Source code and Dockerfiles

ArgoCD Application Structure

Each microservice has a dedicated ArgoCD Application manifest:

# argocd/cart.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cart
  namespace: argocd
spec:
  destination:
    namespace: shopstack    # Target Kubernetes namespace
    server: https://kubernetes.default.svc
  project: default
  source:
    helm:
      parameters:           # Helm value overrides
        - name: hpa.enabled
          value: "true"
        - name: defaults.namespace.name
          value: "shopstack"
      valueFiles:
        - values.yaml        # Helm values file
    path: helm/cart          # Helm chart path in repo
    repoURL: https://github.com/Uj5Ghare/Kops-Microservices-Project.git
    targetRevision: k8s/staging  # 🎯 KEY: Branch to track
  syncPolicy:
    automated:
      prune: true           # Auto-remove resources
      selfHeal: true        # Auto-fix drift

Helm Chart Structure

Each service has a standardized Helm chart:

# helm/cart/values.yaml
defaults:
  namespace:
    name: shopstack
deploy:
  replicaCount: 1
  image:
    repository: ghcr.io/shopstack/shopstack-cart
    tag: latest              # 🔄 Updated by CI/CD
  tolerations:
    key: app
    operator: Equal
    value: shopstack        # 🎯 Standardized tolerations
  env:
    - name: REDIS_HOST
      value: "redis"
    - name: CATALOGUE_HOST
      value: "catalogue"

service:
  type: ClusterIP
  ports:
    http:
      port: 8080
      targetPort: 8080

hpa:
  enabled: true
  maxReplicas: 3
  minReplicas: 1

🚀 Deployment Workflow with k8s/staging

Step 1: Code Changes & CI/CD

# Developer makes changes
git checkout -b feature/new-feature
# ... make changes ...
git commit -m "Add new feature"
git push origin feature/new-feature

# CI/CD Pipeline (buildspec.yml) triggers:
# 1. Detects changed service folder
# 2. Builds Docker image
# 3. Pushes to ECR with new tag
# 4. Updates Helm values with new image tag
# 5. Pushes to k8s/staging branch

Step 2: ArgoCD Automatic Deployment

# ArgoCD monitors k8s/staging branch
# Detects changes in helm/[service]/values.yaml
# Automatically deploys to staging environment

Step 3: Environment Promotion

# After staging validation:
git checkout k8s/preprod
git merge k8s/staging
git push origin k8s/preprod

# ArgoCD deploys to production

🎯 Key Components

1. ArgoCD Applications (`argocd/`)

Purpose: Define which services to deploy and how
Target: k8s/staging or k8s/preprod branches
Namespace: All services deploy to shopstack namespace
Sync Policy: Automated with self-healing

2. Helm Charts (`helm/`)

Purpose: Kubernetes resource templates for each service
Structure: Standardized across all services
Configuration: Environment-agnostic defaults
Templates: Deployment, Service, HPA, Ingress, etc.

3. Value Files (`valuefiles/`)

Purpose: Environment-specific configurations
Infrastructure: Monitoring, logging, security tools
Tolerations: Standardized app=shopstack pattern
Integration: ArgoCD, Prometheus, Elasticsearch, etc.

🔄 Complete Deployment Flow

graph LR
    A[Developer Push] --> B[CI/CD Pipeline]
    B --> C[Build Docker Image]
    C --> D[Push to ECR]
    D --> E[Update Helm Values]
    E --> F[Push to k8s/staging]
    F --> G[ArgoCD Detects Changes]
    G --> H[Deploy to Staging]
    H --> I[Testing & Validation]
    I --> J[Manual Approval]
    J --> K[Merge to k8s/preprod]
    K --> L[ArgoCD Deploy to Production]

📊 Service Dependencies

# ArgoCD Application Dependencies:
web → catalogue, user, shipping, payment, ratings
user → mongodb, redis
catalogue → mongodb
cart → redis
payment → rabbitmq
shipping → mysql
ratings → mysql
dispatch → rabbitmq

🎛️ Configuration Management

Environment Variables (per service):

# Cart Service
REDIS_HOST: redis
CATALOGUE_HOST: catalogue
INSTANA_AUTO_PROFILE: true

# User Service  
REDIS_HOST: redis
MONGO_URL: mongodb://mongodb:27017/users

Standardized Settings (all services):

tolerations:
  key: app
  operator: Equal  
  value: shopstack

affinity:
  nodeAffinity:
    key: app
    operator: In
    values: shopstack

🚀 Benefits of This Structure

GitOps: Everything is code-controlled and versioned
Environment Isolation: Separate branches for staging/production
Automated Deployments: ArgoCD handles deployment automatically
Standardization: Consistent structure across all services
Scalability: Easy to add new services following the same pattern
Rollback: Simple branch-based rollback capability
Observability: Integrated monitoring and logging stack

Cluster Management

Tool: Kubernetes Operations (kops)
Environments:
- Staging: k8s/staging branch
- Production: k8s/preprod branch

Node Tolerations

All services use standardized tolerations:

tolerations:
  key: app
  operator: Equal
  value: shopstack

ArgoCD Configuration

GitOps: Automated deployments from Git
Helm Charts: Dynamic value updates
Sync Strategy: Automated with manual overrides
Health Monitoring: Application and cluster health

Namespace Strategy

Unified Namespace: shopstack for all services
Isolation: Environment-specific deployments
Resource Management: Centralized resource quotas

📊 Monitoring & Observability

Logging Stack

Fluentd/Fluentbit: Log collection and forwarding
Elasticsearch: Centralized log storage and indexing
Kibana: Log visualization and analysis dashboard
CloudWatch: AWS log aggregation backup

Metrics & Alerting

Prometheus: Metrics collection from all services
AlertManager: Alert routing and management
Custom Alerts: Application-specific monitoring rules
Slack Integration: Real-time alert notifications

Distributed Tracing

OpenTelemetry: End-to-end request tracing
Service Mesh: Inter-service communication monitoring
Performance Analysis: Request latency and error tracking
Instana Integration: APM and synthetic monitoring

Health Monitoring

All services expose /health endpoints:

Response Format: JSON with service status
Dependencies: Database and external service checks
Automated Testing: Synthetic transaction monitoring

🔒 Security

Container Security

Base Images: Official and trusted base images
Minimal Privileges: Least-privilege container execution
Secret Management: Kubernetes secrets and environment variables
Image Scanning: Automated vulnerability scanning

Network Security

Service Mesh: Internal network isolation
TLS Encryption: Inter-service communication
API Gateway: Centralized access control
Rate Limiting: DDoS protection and abuse prevention

Data Protection

Encryption at Rest: MongoDB and Redis encryption
Encryption in Transit: TLS for all communications
PII Protection: Personal data anonymization
Compliance: GDPR and PCI DSS considerations

📁 Project Structure

Kops-Microservices-Project/
├── 📁 Service Directories
│   ├── web/           # Frontend application
│   ├── user/          # User management service
│   ├── catalogue/     # Product catalog service
│   ├── cart/          # Shopping cart service
│   ├── payment/       # Payment processing service
│   ├── shipping/      # Shipping fulfillment service
│   ├── ratings/       # Product rating service
│   ├── dispatch/       # Order dispatch service
│   ├── mongo/          # Database configuration
│   └── mysql/          # Database scripts
├── 📁 Infrastructure
│   ├── docker-compose.yaml    # Local development setup
│   ├── buildspec.yml        # CI/CD pipeline
│   └── arch.png           # Architecture diagram
├── 📁 Kubernetes
│   └── valuefiles/         # Helm value files
│       ├── argocd-values.yaml
│       ├── prometheus-values.yaml
│       ├── elasticsearch-values.yaml
│       └── [other service configs]
├── 📁 Configuration
│   ├── .env.example       # Environment template
│   └── .gitignore        # Git ignore rules
└── 📁 Documentation
    ├── README.md           # This file
    └── LICENSE            # Project license

Service Directory Structure

Each service follows the standard structure:

[service]/
├── Dockerfile          # Container build definition
├── package.json        # Node.js dependencies
├── src/               # Application source code
├── helm/              # Kubernetes deployment charts
│   └── values.yaml    # Service configuration
└── tests/              # Unit and integration tests

🚀 Quick Start

1. Local Development

# Clone and setup
git clone https://github.com/Uj5Ghare/Kops-Microservices-Project.git
cd Kops-Microservices-Project
cp .env.example .env

# Edit environment variables with your values
nano .env

# Start all services
docker-compose up -d

# Access services
open http://localhost:8080  # Web frontend

# Check service health
curl http://localhost:8080/health  # Web service
curl http://localhost:8080/health  # Other services (via web proxy)

2. Service-Specific Development

# Build specific service
docker-compose build catalogue

# Start specific service with dependencies
docker-compose up -d mongodb catalogue

# View service logs
docker-compose logs -f catalogue

# Access service directly (if exposed)
curl http://localhost:8080/health  # Web service only

3. Kubernetes Deployment

# Deploy to staging
git checkout k8s/staging
git push origin k8s/staging

# Deploy to production
git checkout k8s/preprod
git push origin k8s/preprod

3. Monitoring Access

Kibana Dashboard: https://observability.ujwal5ghare.xyz
Prometheus: Available via cluster port forwarding
ArgoCD: https://argocd.ujwal5ghare.xyz

🔄 CI/CD Workflow

graph LR
    A[Developer Push] --> B[CodeBuild Trigger]
    B --> C[Build & Test]
    C --> D[Push to ECR]
    D --> E[Update Helm Values]
    E --> F[ArgoCD Sync]
    F --> G[Kubernetes Deploy]
    G --> H[Health Check]
    H --> I[Slack Notification]

🛠️ Troubleshooting

Common Issues

Service Won't Start

# Check logs
docker-compose logs [service-name]

# Verify environment
docker-compose config

# Check port conflicts
netstat -tulpn | grep [port]

Build Failures

# Check build logs
aws codebuild batch-get-builds --project-name shopstack

# Verify ECR permissions
aws ecr get-authorization-token

Deployment Issues

# Check ArgoCD status
kubectl get applications -n argocd

# Verify tolerations
kubectl describe nodes

📚 Additional Resources

Documentation

Support

Issues: GitHub Issues
Discussions: GitHub Discussions

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Fork the repository
Create a feature branch
Make your changes
Add tests if applicable
Submit a pull request

Built with ❤️ for the ShopStack e-commerce platform

Name		Name	Last commit message	Last commit date
Latest commit History 59 Commits
.github/workflows		.github/workflows
cart		cart
catalogue		catalogue
dispatch		dispatch
mongo		mongo
mysql		mysql
payment		payment
ratings		ratings
shipping		shipping
slack		slack
user		user
web		web
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
buildspec.yml		buildspec.yml
docker-compose.yaml		docker-compose.yaml
kops-arch.png		kops-arch.png
shopstack-arch.png		shopstack-arch.png

Folders and files

Latest commit

History

Repository files navigation