Delimiters longer than one character

[maob05]

We have a csv-file that is double semi-colon separated that we read through Sling. Sling interprets every single semicolon as a delimiter by default so we get empty columns and data being "pushed" to the wrong column. How can I get it to see ;; as a delimiter?

[flarco]

Sling doesn't support this natively, but you can use custom SQL for this (which uses duckdb), see https://docs.slingdata.io/examples/file-to-database/custom-sql
Just pass in the delimiter in the read_csv function, as shown here.

[NatarajNalliannan-spp]

Hello Flarco,
I have looked into your suggestion and it sounds like more into duckdb,
Let me explain our scenario, We are loading the csv file from azure blob storage and loading directly into snowflake,
Below is the configuration we use

Default settings for all streams

defaults:
mode: incremental # Incremental mode ensures only new or updated data is processed
object: 'file_name.{stream_file_folder}_{stream_file_name}' # Naming convention for the target object
update_key: _sling_loaded_at # Key used to track updates
columns:
'*': string # cast all columns as string
source_options:
format: csv
header: false # Indicates that the source file does not have a header row

Stream-specific configurations

streams:
#CONCERN
"file_name":
object: "{schema_name}.file_name" # Target object name
single: true
transforms:
"*": [ decode_latin1 ] # Apply decode_latin1 transformation to all columns
meta:
dagster:
description: "data from csv file" # Description for the stream

in this configuration, sling accepts only one character as delimiter, but we have two delimiter as like ;;
is it possible to parse the file with ;; with the above configuration.
we use duckdb in local development to test however the target is snowflake.

your recommendations are really appreciated,
Let me know if we need more info. thanks !

[flarco]

This isn't possible at the moment. Added a feature request: #560

[flarco]

Implemented for next release.