gNMI sync fails with Arista cEOS - "invalid message type: *gnmi.SubscribeRequest"

[reinaldosaraiva]

GitHub Issue - SDC data-server Bug

Repository: https://github.com/sdcio/data-server/issues

---

Title

gNMI sync fails with Arista cEOS - "invalid message type: *gnmi.SubscribeRequest"

---

Labels

bug, gnmi, arista, data-server

---

Issue Body

Description

SDC data-server fails to sync with Arista cEOS using gNMI Subscribe Request. The error occurs with both 4.33.0F and 4.34.1F versions, and with both JSON_IETF and JSON encodings.

RunningConfig remains empty despite Target showing READY=True.

Error Message

{
  "time": "2026-01-08T17:18:54Z",
  "level": "ERROR",
  "msg": "failed to sync target",
  "datastore-name": "sdc.spine-1",
  "logger": "datastore/sync/Sync",
  "err": "option DataType: invalid message type: *gnmi.SubscribeRequest",
  "mode": "once"
}

The error repeats every 2-3 seconds, indicating continuous retry attempts.

---

Environment

SDC Version:

• config-server: v0.0.54
• data-server: v0.0.66
• schema-server: integrated

Target Device:

• Vendor: Arista Networks
• OS: cEOS (containerized EOS)
• Version 1: 4.34.1F (tested)
• Version 2: 4.33.0F (tested)
• Platform: x86_64-kvm_x86_64-r0
• Image: ceos:4.33.0F and ceos:4.34.1F

Deployment:

• Kubernetes: KinD v1.32.0
• Containerlab: v0.71.1
• Network: Docker bridge (172.20.20.0/24)

---

Configuration

TargetConnectionProfile:

apiVersion: inv.sdcio.dev/v1alpha1
kind: TargetConnectionProfile
metadata:
  name: gnmi-eos
  namespace: sdc
spec:
  protocol: gnmi
  port: 6030
  encoding: JSON_IETF
  insecure: true
  skipVerify: true
  timeout: 30s
  connectRetry: 30s
  commitCandidate: candidate
  includeNS: false
  operationWithNS: false
  preferredNetconfVersion: "1.0"
  useOperationRemove: false

TargetSyncProfile:

apiVersion: inv.sdcio.dev/v1alpha1
kind: TargetSyncProfile
metadata:
  name: gnmi-sync
  namespace: sdc
spec:
  sync:
  - mode: once
    paths:
    - "/"
    protocol: gnmi
    port: 6030
  buffer: 0
  workers: 2
  validate: true

Target Status:

$ kubectl get target spine-1 -n sdc
NAME      READY   REASON   PROVIDER               VERSION   ADDRESS
spine-1   True             eos.arista.sdcio.dev   4.33.0F   172.20.20.11

RunningConfig Status:

$ kubectl get runningconfig spine-1 -n sdc -o jsonpath='{.status.value}'
{}  # ← Empty

---

Evidence: gnmic Native Test (CRITICAL)

To isolate whether the issue was in SDC or Arista, we tested gNMI Subscribe directly using gnmic v0.37.0 (OpenConfig official tool).

Test Setup:

• Tool: gnmic v0.37.0
• Target: Same Arista cEOS switch (172.20.20.11:6030)
• Credentials: admin/admin
• Subscribe mode: once (same as SDC)
• Subscribe path: "/" (same as SDC)

Results:

Encoding	gnmic (native)	SDC data-server
JSON_IETF	✅ SUCCESS	❌ FAILS
JSON	✅ SUCCESS	❌ FAILS
PROTO	✅ SUCCESS	❌ N/A

Commands Executed:

# JSON_IETF - SUCCESS
gnmic -a 172.20.20.11:6030 \
  -u admin -p admin \
  --insecure \
  --encoding json_ietf \
  subscribe --path "/" --mode once

# JSON - SUCCESS
gnmic -a 172.20.20.11:6030 \
  -u admin -p admin \
  --insecure \
  --encoding json \
  subscribe --path "/" --mode once

# PROTO - SUCCESS
gnmic -a 172.20.20.11:6030 \
  -u admin -p admin \
  --insecure \
  --encoding proto \
  subscribe --path "/" --mode once

Sample Data Received (truncated):

{
  "source": "172.20.20.11:6030",
  "subscription-name": "default-1767893168",
  "timestamp": 1767892423366941952,
  "time": "2026-01-08T17:13:43.366941952Z",
  "prefix": "system/messages",
  "updates": [
    {
      "Path": "config/severity",
      "values": {
        "config/severity": "DEBUG"
      }
    }
  ]
}

Paths Successfully Retrieved:

• system/messages
• system/ssh-server/state/counters
• system/console/state/counters
• arista/redundancy/management-active
• system/aaa/authentication/users/user[username=admin]/state
• Hundreds of other paths (truncated for brevity)

---

Conclusion

This proves the issue is in SDC data-server, NOT in Arista EOS:

✅ Arista cEOS gNMI implementation works perfectly
✅ All encodings supported (JSON_IETF, JSON, PROTO)
✅ Subscribe mode "once" works correctly
✅ Path "/" returns complete data

❌ SDC data-server constructs gNMI SubscribeRequest incorrectly
❌ Error mentions "option DataType: invalid message type"
❌ gnmic does NOT send this "DataType" option and works fine

---

Hypothesis

The error message "option DataType: invalid message type: *gnmi.SubscribeRequest" suggests that:

1. SDC is adding a custom option called "DataType" to the gNMI SubscribeRequest
2. Arista EOS validates this option and rejects it as invalid
3. gnmic does NOT send this option (or sends it correctly) and succeeds

gNMI Specification Reference:

According to the gNMI proto spec, SubscribeRequest should use Encoding field in SubscriptionList, not a custom "DataType" option:

message SubscriptionList {
  repeated Subscription subscription = 1;
  // ...
  Encoding encoding = 4;  // ← Standard field
}

---

Steps to Reproduce

1. Deploy Arista cEOS 4.33.0F or 4.34.1F using containerlab
2. Enable gNMI on port 6030 with admin/admin credentials
3. Install SDC v0.0.54+ in Kubernetes cluster
4. Create TargetConnectionProfile with encoding=JSON_IETF
5. Create TargetSyncProfile with mode=once, paths=["/"]
6. Create Target pointing to Arista switch
7. Observe logs: kubectl logs -n sdc config-server-XXX -c data-server

Expected Behavior:

• RunningConfig should be populated with switch state
• No errors in data-server logs

Actual Behavior:

• RunningConfig remains empty: {}
• Error repeats every 2-3s: "option DataType: invalid message type"

---

Workaround

Use gnmic directly for testing/validation:

gnmic -a <switch-ip>:6030 \
  -u admin -p admin \
  --insecure \
  --encoding json_ietf \
  subscribe --path "/" --mode once

For production, consider using Ansible with arista.eos collection until this bug is fixed.

---

Impact

Severity: HIGH

• ❌ Blocks ConfigSets (Phase 3) - cannot apply declarative configurations
• ❌ Blocks GitOps workflows
• ❌ RunningConfig always empty
• ❌ Affects ALL Arista cEOS versions tested (4.33.0F, 4.34.1F)
• ❌ Affects multiple encodings (JSON_IETF, JSON)

Affected Users:

• Anyone using SDC with Arista cEOS
• Anyone using SDC with Arista hardware switches (potentially)

---

Additional Information

Tests Performed:

1. ✅ Encoding change (JSON_IETF → JSON): Did NOT resolve
2. ✅ EOS version downgrade (4.34.1F → 4.33.0F): Did NOT resolve
3. ✅ Schema update (using official 4.33.0F schema): Did NOT resolve
4. ✅ gnmic native test: RESOLVED - proves bug is in SDC

Time Invested:

• ~12 hours of systematic troubleshooting
• Multiple hypotheses tested and ruled out
• Root cause definitively isolated to SDC data-server

Documentation:

• Complete troubleshooting process documented
• All configurations tested and validated
• Evidence collected for easy reproduction

---

Logs

SDC data-server Error (repeating):

{"time":"2026-01-08T17:18:54Z","level":"ERROR","msg":"failed to sync target","datastore-name":"sdc.spine-1","logger":"datastore/sync/Sync","err":"option DataType: invalid message type: *gnmi.SubscribeRequest","mode":"once"}
{"time":"2026-01-08T17:18:57Z","level":"ERROR","msg":"failed to sync target","datastore-name":"sdc.spine-1","logger":"datastore/sync/Sync","err":"option DataType: invalid message type: *gnmi.SubscribeRequest","mode":"once"}
{"time":"2026-01-08T17:19:00Z","level":"ERROR","msg":"failed to sync target","datastore-name":"sdc.spine-1","logger":"datastore/sync/Sync","err":"option DataType: invalid message type: *gnmi.SubscribeRequest","mode":"once"}

gnmic Success (sample):

$ gnmic -a 172.20.20.11:6030 -u admin -p admin --insecure \
    --encoding json_ietf subscribe --path "/" --mode once

{
  "source": "172.20.20.11:6030",
  "subscription-name": "default-1767893168",
  "timestamp": 1767892423366941952,
  "time": "2026-01-08T17:13:43.366941952Z",
  "prefix": "system/messages",
  "updates": [
    {
      "Path": "config/severity",
      "values": {
        "config/severity": "DEBUG"
      }
    }
  ]
}
[... hundreds more paths successfully received ...]

---

Suggested Fix

Investigation needed in data-server code:

1. Search for where gNMI SubscribeRequest is constructed
2. Check if a custom "DataType" option is being added
3. Compare with gnmic implementation (which works)
4. Remove or fix the "DataType" option to match gNMI spec

Code locations to check:

# In data-server repository
grep -r "DataType" --include="*.go"
grep -r "SubscribeRequest" --include="*.go"
grep -r "SubscriptionList" --include="*.go"

---

Related Issues

• (Search for similar Arista/gNMI issues in sdcio repos)

---

References

• gNMI Specification: https://github.com/openconfig/gnmi/blob/master/proto/gnmi/gnmi.proto
• gnmic Tool: https://github.com/openconfig/gnmic
• Arista YANG Models: https://github.com/aristanetworks/yang
• SDC Documentation: https://docs.sdcio.dev

---

Attachments

(When creating the actual issue, attach these files):

1. data-server-logs.txt - Complete logs showing repeated errors
2. gnmic-success-output.txt - Full gnmic output showing success
3. target-config.yaml - Complete Target/Profile configurations
4. target-status.yaml - kubectl get target -o yaml output

---

Contact

Available for:

• Providing additional logs/configs
• Testing patches
• Validating fixes
• Contributing PR if pointed to the right code location

---

Thank you for maintaining SDC! This is a great project and we're eager to see Arista support working properly.

[reinaldosaraiva]

Attachments

Complete evidence files for reproduction and validation.

[reinaldosaraiva]

data-server-logs.txt:

[reinaldosaraiva]

target-config.yaml:

[reinaldosaraiva]

target-status.yaml:

[reinaldosaraiva]

gnmic-success-output.txt (3.3 MB):

Full gnmic output showing successful gNMI Subscribe with all encodings:

[reinaldosaraiva]

gnmic-success-output.txt (3.3 MB):

Full gnmic output showing successful gNMI Subscribe with all encodings (JSON_IETF, JSON, PROTO):
https://gist.github.com/reinaldosaraiva/665f29e228b11f9fda97657c7133c93c

This proves Arista cEOS gNMI implementation works perfectly - the bug is in SDC data-server.

[henderiw]

Thx for the detailed logs. we are just releasing the sync code, so far we support get, but started to add dynamic sync. On Change mainly. @steiler will know if we also do ONCE as you have configured, but we typically need updates to do revertive/non revertive behavior.

Could you try the get method, maybe ?

subscription modes:

get: Periodic get Config, interval needs to be specified.
onChange: Corresponds to gNMI stream mode ON_CHANGE.
sample: Matches gNMI stream mode SAMPLE, using interval as the sample-interval.
once: Equivalent to gNMI mode ONCE, where a SubscribeRequest is sent at each interval.

We are about to release the sync code, but @steiler can shine in as well

[steiler]

@reinaldosaraiva you are perfectly right.
Taking a look here, we add the option:
https://github.com/sdcio/data-server/blob/main/pkg/datastore/target/gnmi.go#L432

But it is clear, from the gnmic docs, that the option is only for GetRequests:
https://github.com/openconfig/gnmic/blob/main/pkg/api/gnmi_msgs.go#L691-L692

But also as @henderiw mentioned, we ran for quite some time now with the GNMI Get as the main and only properly supported option. However, recently we've started reworking the whole GNMI sync to bring back the streaming options.

We will soon merge #355 which contains these capabilities in a sort of alpha version. We need to get the integration tests setup for these to be able to claim them ready for use.
As @henderiw mentioned, try using the Get Sync for now from the released version or feel free to build a container yourself from the PR code and test it. All possible feedback is welcome.

Please also join our discord channel and send me a PM or get in touch in the public channels. I'll be able to assist and also provide early feedback for any questions or issues you might have.

https://discord.com/channels/1240272304294985800/1311031796372344894

[reinaldosaraiva]

Update: GET mode partially resolves the DataType bug ✅

Thank you @henderiw and @steiler for the quick response and confirmation!

Test Results

Changed configuration:

apiVersion: inv.sdcio.dev/v1alpha1
kind: TargetSyncProfile
metadata:
  name: gnmi-sync
  namespace: sdc
spec:
  sync:
    - name: full-sync
      protocol: gnmi
      port: 6030
      paths: [/]
      mode: get        # ← Changed from "once"
      interval: 60s    # ← Added interval

Results After Restart

✅ DataType Error Resolved:

• Previous error: "option DataType: invalid message type: *gnmi.SubscribeRequest","mode":"once"
• After change to mode: get: DataType error disappeared completely
• config-server restart was required to reload the profile

New Issue Discovered

⚠️ Schema parsing error now appears:

{"time":"2026-01-08T20:08:25Z","level":"ERROR","msg":"failed to write to sync tree",
 "datastore-name":"sdc.spine-1","logger":"datastore/sync",
 "err":"unknown object \"name\" under container \"config\""}

{"time":"2026-01-08T20:08:25Z","level":"ERROR","msg":"failed exporting tree",
 "datastore-name":"sdc.spine-1","logger":"datastore/sync",
 "err":"intent not present"}

Targets status:

NAME      READY   REASON                
spine-1   False   datastore not ready
leaf-1    False   datastore not ready   
leaf-2    False   datastore not ready

Analysis

1. GET mode successfully resolves the DataType bug ✅
2. The schema parsing error may be unrelated to the DataType issue
3. Possibly related to Arista cEOS 4.33.0F schema compatibility

Environment Details

• SDC Version: v0.0.54
• Device: Arista cEOS 4.33.0F (containerlab)
• Schema: eos.arista.4.33.0f (3-repo pattern: aristanetworks/yang + YangModels/yang + sdcio/arista-yang-patch)
• TargetConnectionProfile: gnmi, port 6030, encoding JSON_IETF
• Native gNMI test: ✅ Works perfectly with gnmic v0.37.0

Next Steps

Will join the Discord channel as suggested to debug the schema parsing error. The DataType bug fix is confirmed working with GET mode.

Thank you for maintaining this excellent project! 🙏

[steiler]

I can confirm this issue.
However it is not a schema issue, If you take a look at the schema CR, that is loading perfectly fine.
I'll keep you posted on my findings.