Elasticsearch MCP Server

Create a powerful Model Context Protocol (MCP) server for Elasticsearch in minutes with our AI Gateway. This guide walks you through setting up seamless search and analytics integration with enterprise-grade capabilities and instant API authentication.

About Elasticsearch API

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. The API provides comprehensive access to indexing, searching, and analyzing large volumes of data in near real-time.

Key Capabilities

• Full-Text Search: Advanced text analysis and search
• Structured Queries: Complex filtering and aggregations
• Analytics Engine: Real-time data analytics
• Distributed Architecture: Horizontal scaling
• Multi-tenancy: Index-level isolation
• Near Real-time: Millisecond search latency
• Schema-free: Dynamic mapping
• RESTful API: JSON over HTTP

API Features

• Document APIs: CRUD operations on documents
• Search APIs: Query DSL for complex searches
• Aggregations: Analytics and metrics
• Index APIs: Manage indices and mappings
• Cluster APIs: Monitor and manage clusters
• Ingest Pipelines: Data transformation
• Security: Authentication and authorization
• Machine Learning: Anomaly detection and forecasting

What You Can Do with Elasticsearch MCP Server

The MCP server transforms Elasticsearch API into a natural language interface, enabling AI agents to:

Search Operations

• Full-Text Search

  • "Search for documents containing 'artificial intelligence'"
  • "Find products with 'wireless' in description"
  • "Search logs for error messages"
  • "Query articles about machine learning"

• Filtered Searches

  • "Find orders from last week over $100"
  • "Search users in California"
  • "Get products in stock under $50"
  • "Find logs between specific dates"

• Fuzzy Matching

  • "Search for 'intelligense' (with typos)"
  • "Find similar product names"
  • "Match approximate values"
  • "Phonetic searches"

Analytics and Aggregations

• Metrics

  • "Calculate average order value"
  • "Sum total revenue by month"
  • "Count unique visitors"
  • "Get min/max prices"

• Bucket Aggregations

  • "Group sales by category"
  • "Histogram of prices"
  • "Date histogram of events"
  • "Terms aggregation by status"

• Pipeline Aggregations

  • "Calculate moving averages"
  • "Compute cumulative sums"
  • "Derivative calculations"
  • "Bucket comparisons"

Document Management

• CRUD Operations

  • "Index new document"
  • "Update product details"
  • "Delete old records"
  • "Bulk import data"

• Versioning

  • "Track document versions"
  • "Optimistic concurrency control"
  • "Conflict resolution"
  • "Version history"

Index Management

• Index Operations

  • "Create new index"
  • "Delete old indices"
  • "Reindex data"
  • "Clone index"

• Mapping Management

  • "Define field types"
  • "Update mappings"
  • "Add new fields"
  • "Configure analyzers"

• Index Templates

  • "Create index patterns"
  • "Auto-apply settings"
  • "Lifecycle policies"
  • "Rollover indices"

Cluster Management

• Health Monitoring

  • "Check cluster health"
  • "Monitor node status"
  • "View shard allocation"
  • "Track performance metrics"

• Configuration

  • "Update cluster settings"
  • "Manage node roles"
  • "Configure allocation"
  • "Set routing rules"

Advanced Features

• Geospatial Queries

  • "Find stores within 10km"
  • "Search by polygon area"
  • "Calculate distances"
  • "Geo aggregations"

• Machine Learning

  • "Detect anomalies in metrics"
  • "Forecast future trends"
  • "Classify documents"
  • "Find outliers"

• Percolator Queries

  • "Match documents to queries"
  • "Real-time alerting"
  • "Saved search monitoring"
  • "Reverse search"

Prerequisites

• Access to Cequence AI Gateway
• Elasticsearch cluster (self-hosted or cloud)
• Cluster endpoint URL
• Authentication credentials (if secured)

Step 1: Configure Elasticsearch Access

1.1 Cluster Connection

1. Note your Elasticsearch endpoint:
   • Self-hosted: http://localhost:9200
   • Elastic Cloud: https://cluster-id.region.cloud-provider.elastic-cloud.com
2. Ensure cluster is accessible
3. Test connection with curl

1.2 Authentication Setup

For secured clusters, Elasticsearch supports multiple authentication methods:

1. Basic Auth: Username and password
2. API Keys: Generate from Kibana
3. Cloud ID: For Elastic Cloud deployments
4. Certificates: For TLS/SSL connections
5. OAuth 2.0 using external providers (e.g., Okta): Enterprise SSO integration
6. JWT (JSON Web Tokens): Token-based authentication

1.3 Security Configuration

Basic API Key Setup

# Generate API key
POST /_security/api_key
{
  "name": "ai-gateway-key",
  "role_descriptors": {
    "ai_gateway_role": {
      "cluster": ["monitor", "manage"],
      "index": [
        {
          "names": ["*"],
          "privileges": ["all"]
        }
      ]
    }
  }
}

OIDC Authentication with Okta

For enterprise deployments requiring Single Sign-On (SSO), configure OIDC authentication with Okta:

Step 1: Configure Okta Application

1. Create a new Web Application in Okta
2. Configure the following settings:
   • Grant Types: Authorization Code, Refresh Token, Client Credentials
   • Sign-in redirect URIs:
     • https://your-kibana-url/api/security/oidc/callback
     • https://auth.aigateway.cequence.ai/v1/outbound/oauth/callback
   • Sign-out redirect URIs: http://localhost:8080
3. Note your Client ID (e.g., your-client-id)
4. Generate and save Client Secret

Step 2: Configure Elasticsearch User Settings

1. Log in to Elastic Cloud console
2. Navigate to your deployment
3. Click Edit in the left navigation menu
4. Scroll down to the Elasticsearch section
5. Click Edit user settings to modify elasticsearch.yml
6. Add OIDC realm configuration:

xpack.security.authc.realms.oidc.oidc1:
  order: 10
  rp.client_id: "your-client-id"
  rp.response_type: "code"
  rp.redirect_uri: "https://your-kibana-url/api/security/oidc/callback"
  op.issuer: "https://your-okta-domain.okta.com"
  op.authorization_endpoint: "https://your-okta-domain.okta.com/oauth2/v1/authorize"
  op.token_endpoint: "https://your-okta-domain.okta.com/oauth2/v1/token"
  op.jwkset_path: "https://your-okta-domain.okta.com/oauth2/v1/keys"
  op.userinfo_endpoint: "https://your-okta-domain.okta.com/oauth2/v1/userinfo"
  op.endsession_endpoint: "https://your-okta-domain.okta.com/oauth2/v1/logout"
  claims.principal: "sub"
  claims.groups: "groups"

7. Click Save at the bottom of the page to apply changes and restart the deployment

Step 3: Configure JWT Realm

Configure JWT realm for API authentication:

xpack.security.authc.realms.jwt.jwt1:
  order: 2
  client_authentication.type: none
  allowed_issuer: "https://your-okta-domain.okta.com/oauth2/default"
  allowed_audiences: ["api://default", "your-client-id"]
  allowed_signature_algorithms: ["RS256"]
  pkc_jwkset_path: "https://your-okta-domain.okta.com/oauth2/default/v1/keys"
  claims.principal: "sub"
  claims.groups: "groups"

Step 4: Store Client Secret in Elasticsearch Keystore

# Add client secret to keystore
bin/elasticsearch-keystore add xpack.security.authc.realms.oidc.oidc1.rp.client_secret

Step 5: Configure Role Mappings

Map Okta users to Elasticsearch roles to grant access:

1. Open Kibana dev tools or use curl
2. Create a role mapping for the OIDC realm:

PUT /_security/role_mapping/oidc_kibana_mapping
{
  "enabled": true,
  "roles": ["superuser"],
  "rules": {
    "all": [
      { "field": { "realm.name": "oidc1" } },
      { "field": { "groups": "everyone" } }
    ]
  },
  "metadata": {
    "version": 1
  }
}

Note: Adjust the rules to match your specific Okta groups or user claims.

Step 6: Configure Okta Access Policy

1. In Okta, create an Access Policy:

   • Name: "Your Access Policy Name" (e.g., "Gateway Access Policy")
   • Description: "Your policy description" (e.g., "Allow Gateway to access APIs")
   • Assigned to: All Clients or specific clients

2. Click Add Rule and configure the following:

   IF Grant type is:

   • Select Authorization Code (required for OAuth 2.0 flow)
   • Optionally select Client Credentials if needed for API access

   AND User is:

   • Any user assigned the app, or
   • Assigned the app and a member of one of the following: (select specific groups if needed)

   AND Scopes requested:

   • Any scopes, or
   • The following scopes: (specify: openid, email, profile)

   THEN Use this inline hook:

   • None (disabled) (default)

   AND Access token lifetime is:

   • 1 Hours (default, adjust as needed)

   AND Refresh token lifetime is:

   • 90 Days (default, adjust as needed)
   • but will expire if not used every: 7 Days (default, adjust as needed)

3. Enter a Rule Name (e.g., "Allow Authorization Code")

4. Set Status: Active

5. Click Create Rule

Step 7: Create MCP Server

Navigate to the AI Gateway, go to Apps > Third-party > Elasticsearch, and click Create MCP Server.

Step 8: Configure API Endpoints

1. Base URL: Your Elasticsearch endpoint
2. Click Select Endpoints to choose which API operations to enable
3. Click Next

Step 9: MCP Server Configuration

1. Name: "Elasticsearch Search"
2. Description: "Search and analytics engine"
3. Configure security settings
4. Choose deployment options
5. Click Next

Step 10: Configure Authentication

1. Authentication Type: Select OAuth 2.0
2. Configure the Okta details:
   • Authorization URL: https://your-okta-domain.okta.com/oauth2/default/v1/authorize
   • Token URL: https://your-okta-domain.okta.com/oauth2/default/v1/token
   • Client ID: your-client-id
   • Client Secret: Your generated secret
   • Redirect URI: https://auth.aigateway.cequence.ai/v1/outbound/oauth/callback
   • Scopes: openid, email, profile
3. Click Connect to test the authentication
4. Click Create

Available Elasticsearch API Operations

Search APIs

• Search

  • Query DSL
  • Request body search
  • URI search
  • Multi-index search

• Multi Search

  • Batch queries
  • Different indices
  • Optimized performance
  • Reduced overhead

• Scroll & Search After

  • Large result sets
  • Pagination
  • Point in time
  • Consistent results

Document APIs

• Index/Create

  • Add documents
  • Auto-generate IDs
  • Versioning
  • Refresh control

• Get/Multi-Get

  • Retrieve by ID
  • Source filtering
  • Stored fields
  • Batch retrieval

• Update/Delete

  • Partial updates
  • Scripts
  • Delete by query
  • Bulk operations

Index APIs

• Create Index

  • Settings
  • Mappings
  • Aliases
  • Lifecycle

• Index Management

  • Open/Close
  • Freeze/Unfreeze
  • Shrink/Split
  • Rollover

Aggregation Types

• Metrics: sum, avg, min, max, stats
• Bucket: terms, histogram, date_histogram, range
• Pipeline: moving_avg, cumulative_sum, bucket_script
• Matrix: stats, covariance, correlation

Step 11: Review and Deploy

1. Review all configuration settings
2. Click Deploy to create your MCP server

Using Your Elasticsearch MCP Server

Setup Instructions:

• Claude Desktop
• Cursor
• Windsurf

Natural Language Commands

• "Search for products containing 'laptop' under $1000"
• "Show me error logs from the last hour"
• "Calculate average response time by endpoint"
• "Find top 10 customers by order count"
• "Analyze sentiment in customer reviews"

Common Use Cases

Log Analytics

• Centralized logging
• Error tracking
• Performance monitoring
• Security analysis

E-commerce Search

• Product search
• Faceted navigation
• Recommendation engine
• Inventory tracking

Full-Text Search

• Document search
• Knowledge base
• Content discovery
• Semantic search

Real-time Analytics

• Metrics aggregation
• Dashboard data
• Trend analysis
• Anomaly detection

Best Practices

1. Index Design:

   • Plan mappings carefully
   • Use appropriate field types
   • Consider search requirements
   • Optimize for your queries

2. Query Optimization:

   • Use filters for exact matches
   • Leverage caching
   • Limit result size
   • Use source filtering

3. Performance Tuning:

   • Monitor cluster health
   • Configure appropriate shards
   • Use bulk operations
   • Implement pagination

Troubleshooting

Common Issues

1. Connection Errors

   • Verify cluster URL
   • Check network connectivity
   • Validate certificates
   • Test authentication

2. Query Performance

   • Review query complexity
   • Check index size
   • Monitor resource usage
   • Optimize mappings

3. Data Issues

   • Verify mapping conflicts
   • Check data types
   • Handle null values
   • Review analyzers

Getting Help

• Documentation: AI Gateway Docs
• Support: support@cequence.ai
• Elastic Docs: elastic.co/guide

---