"colums" YAML key not working (or maybe there is a trick to use it properly?) #642

igrpon · 2025-09-15T08:15:00Z

igrpon
Sep 15, 2025

Hi!

as per docs my understanding columns key is intended to explain sling how to cast data on the way from source to target:
ref: columns docs
ref: columns references in github

My intention was to read CSV file and save column where the date is recorded in format YYYYMMDD as string into target database (postgres in my case), by default sling saves that as bigint or timestamp value (depending how successful type identification was done).

I've tried different combinations of settings (including cli keys and YAML), however it seems that sling ignores its.

In the debug output I see the line:

2025-09-15 11:10:56 DBG casting column 'date' as 'string'

so it looks the config option is taken, however there is any effect on that - the column in db is created with "identified" datatype anyway:

2025-09-15 11:10:56 DBG create table if not exists "moex"."iap_quotes_imoex_tmp" ("ticker" varchar(65500),
  "date" timestamp,
  "close" numeric)

my YAML config:

source: LOCAL
target: POSTGRES

streams:
  "file://./IMOEX_1min_01012025_01092025.csv":
    object: "moex.iap_quotes_imoex"
    mode: full-refresh
    select: [ticker, date, close]

    columns:
      date: string(20)

    source_options:
      format: csv
      delimiter: ";"
      header: true
      encoding: utf8

    target_options:
      column_casing: lower
      adjust_column_type: true

env:
  SLING_SAMPLE_SIZE: 2000

source file:

TICKER;PER;DATE;TIME;HIGH;LOW;CLOSE;VOL
IMOEX;1;2025-01-03 00:00:00;00:00:00;2883.04;2883.04;2883.04;0
IMOEX;1;2025-01-03 10:00:00;10:00:00;2898.71;2888.86;2889.41;0
IMOEX;1;2025-01-03 10:01:00;10:01:00;2889.67;2885.07;2888.47;0
IMOEX;1;2025-01-03 10:02:00;10:02:00;2889.43;2886.5;2888.28;0
...

PS. sling logs (full trace):

2025-09-15 11:07:27 INF Sling CLI | https://slingdata.io
2025-09-15 11:07:28 DBG opened "file" connection (conn-file-ZBw)
2025-09-15 11:07:28 TRC object format map: {"DD":"15","DDD":"15D","HH":"11","MM":"09","MMM":"Sep","YY":"25","YYYY":"2025","hh":"11","mm":"07","object":{"full_name":"\"moex\".\"iap_quotes_imoex\"","name":"\"moex\".\"iap_quotes_imoex\"","schema":"moex","table":"iap_quotes_imoex"},"object_full_name":"\"moex\".\"iap_quotes_imoex\"","object_name":"\"moex\".\"iap_quotes_imoex\"","object_schema":"moex","object_table":"iap_quotes_imoex","run_timestamp":"2025_09_15_110728","source":{"kind":"file","name":"local","type":"file"},"source_kind":"file","source_name":"local","source_type":"file","ss":"28","stream":{"file_ext":"csv","file_folder":".","file_name":"IMOEX_1min_01012025_01092025","file_path":"./IMOEX_1min_01012025_01092025.csv","full_name":"file://./IMOEX_1min_01012025_01092025.csv","name":"file://./imoex_1min_01012025_01092025.csv"},"stream_file_ext":"csv","stream_file_folder":".","stream_file_name":"IMOEX_1min_01012025_01092025","stream_file_path":"./IMOEX_1min_01012025_01092025.csv","stream_full_name":"file://./IMOEX_1min_01012025_01092025.csv","stream_name":"file://./imoex_1min_01012025_01092025.csv","target":{"kind":"database","name":"postgres_fintech","table":"iap_quotes_imoex","type":"postgres"},"target_kind":"database","target_name":"postgres_fintech","target_table":"iap_quotes_imoex","target_type":"postgres","timestamp":{"DD":"15","DDD":"15D","HH":"11","MM":"09","MMM":"Sep","YY":"25","YYYY":"2025","date":"2025-09-15","datetime":"2025-09-15 11:07:28","file_name":"2025_09_15_110728","hh":"11","mm":"07","rfc3339":"2025-09-15T11:07:28+03:00","ss":"28"}}
2025-09-15 11:07:28 TRC len(selectStreams) = 0, len(matchedStreams) = 0, len(replication.Streams) = 1
2025-09-15 11:07:28 INF Sling Replication | LOCAL -> POSTGRES_FINTECH | file://./IMOEX_1min_01012025_01092025.csv
2025-09-15 11:07:28 TRC srcFileProvided: true, tgtFileProvided: false, srcDbProvided: false, tgtDbProvided: true, srcApiProvided: false, srcStreamProvided: true
2025-09-15 11:07:28 TRC using Config:
{
  "source": {
    "conn": "LOCAL",
    "type": "file",
    "stream": "file://./IMOEX_1min_01012025_01092025.csv",
    "select": [
      "ticker",
      "date",
      "close"
    ],
    "primary_key": [],
    "options": {
      "empty_as_null": false,
      "header": true,
      "fields_per_rec": -1,
      "compression": "auto",
      "format": "csv",
      "null_if": "NULL",
      "datetime_format": "AUTO",
      "skip_blank_lines": false,
      "delimiter": ";",
      "max_decimals": -1,
      "encoding": "utf8"
    }
  },
  "target": {
    "conn": "POSTGRES_FINTECH",
    "type": "postgres",
    "object": "\"moex\".\"iap_quotes_imoex\"",
    "columns": [
      {
        "name": "date",
        "type": "string(20)"
      }
    ],
    "options": {
      "datetime_format": "auto",
      "file_max_rows": 0,
      "max_decimals": -1,
      "use_bulk": true,
      "add_new_columns": true,
      "adjust_column_type": false,
      "column_casing": "lower"
    }
  },
  "mode": "full-refresh",
  "options": {},
  "env": {
    "SLING_SAMPLE_SIZE": "2000"
  },
  "stream_name": "file://./IMOEX_1min_01012025_01092025.csv",
  "replication_stream": {
    "mode": "full-refresh",
    "object": "\"moex\".\"iap_quotes_imoex\"",
    "select": [
      "ticker",
      "date",
      "close"
    ],
    "source_options": {
      "header": true,
      "format": "csv",
      "delimiter": ";",
      "encoding": "utf8"
    },
    "target_options": {
      "column_casing": "lower"
    },
    "single": false,
    "columns": [
      {
        "name": "date",
        "type": "string(20)"
      }
    ],
    "hooks": {}
  },
  "incremental_val": null,
  "incremental_val_str": ""
}
2025-09-15 11:07:28 DBG Sling version: 1.4.19 (windows amd64)
2025-09-15 11:07:28 DBG type is file-db
2025-09-15 11:07:28 DBG using: {"columns":[{"name":"date","type":"string(20)"}],"mode":"full-refresh","select":["ticker","date","close"],"transforms":null}
2025-09-15 11:07:28 DBG using source options: {"empty_as_null":false,"header":true,"fields_per_rec":-1,"compression":"auto","format":"csv","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"delimiter":";","max_decimals":-1,"encoding":"utf8"}
2025-09-15 11:07:28 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"lower"}
2025-09-15 11:07:28 INF connecting to target database (postgres)
2025-09-15 11:07:28 TRC driver=postgres conn_url=postgresql://user:pass@127.0.0.1:5432/demo?sslmode=disable
2025-09-15 11:07:28 DBG opened "postgres" connection (conn-postgres-5j1)
2025-09-15 11:07:28 INF reading from source file system (file)
2025-09-15 11:07:28 DBG opened "file" connection (conn-file-nib)
2025-09-15 11:07:28 TRC listing path: file://./IMOEX_1min_01012025_01092025.csv
2025-09-15 11:07:28 DBG reading single datastream from file://./IMOEX_1min_01012025_01092025.csv [format=csv]
2025-09-15 11:07:28 DBG merging csv readers of 1 files [concurrency=3] from file://./IMOEX_1min_01012025_01092025.csv
2025-09-15 11:07:28 DBG processing reader from file://./IMOEX_1min_01012025_01092025.csv
2025-09-15 11:07:28 TRC number-cols detected: 8
2025-09-15 11:07:28 TRC new ds.Start ds.1757923648081.72C
2025-09-15 11:07:28 TRC first row of ds.1757923648081.72C => []interface {}{"IMOEX", "1", "2025-01-03 00:00:00", "00:00:00", "2883.04", "2883.04", "2883.04", "0"}
2025-09-15 11:07:28 TRC ticker - text {"min_len":5,"max_len":5,"min":0,"max":-9223372036854774808,"null_cnt":0,"string_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC per - bigint {"min_len":1,"max_len":1,"min":1,"max":1,"null_cnt":0,"int_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC date - datetime {"min_len":19,"max_len":19,"min":0,"max":-9223372036854774808,"null_cnt":0,"date_cnt":2,"datetime_cnt":898,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC time - text {"min_len":8,"max_len":8,"min":0,"max":-9223372036854774808,"null_cnt":0,"string_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC high - decimal {"min_len":4,"max_len":7,"max_dec_len":2,"min":2795,"max":2898,"null_cnt":0,"dec_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC low - decimal {"min_len":4,"max_len":7,"max_dec_len":2,"min":2794,"max":2888,"null_cnt":0,"dec_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC close - decimal {"min_len":4,"max_len":7,"max_dec_len":2,"min":2794,"max":2889,"null_cnt":0,"dec_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC vol - bigint {"min_len":1,"max_len":1,"min":0,"max":0,"null_cnt":0,"int_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 DBG applying column casing (lower) for target type (postgres)
2025-09-15 11:07:28 DBG casting column 'date' as 'string'
2025-09-15 11:07:28 TRC ds.1757923648081.72C.StreamURL => file://./IMOEX_1min_01012025_01092025.csv
2025-09-15 11:07:28 TRC new batch ds.1757923648081.72C-0
2025-09-15 11:07:28 TRC new ds.Start ds.1757923648112.slb
2025-09-15 11:07:28 TRC first row of ds.1757923648112.slb => []interface {}{"IMOEX", "2025-01-03 00:00:00", "2883.04"}
2025-09-15 11:07:28 TRC ticker - string {"min_len":5,"max_len":5,"min":0,"max":-9223372036854774808,"null_cnt":0,"string_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC date - datetime {"min_len":19,"max_len":19,"min":0,"max":-9223372036854774808,"null_cnt":0,"date_cnt":2,"datetime_cnt":898,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC close - decimal {"min_len":4,"max_len":7,"max_dec_len":2,"min":2794,"max":2889,"null_cnt":0,"dec_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC new batch ds.1757923648112.slb-0
2025-09-15 11:07:28 TRC 1 datastreams pushed [ds.1757923648112.slb]
2025-09-15 11:07:28 TRC pushed 1 datastreams
2025-09-15 11:07:28 TRC []string{"ticker [string]", "date [datetime]", "close [decimal]"}
2025-09-15 11:07:28 INF writing to target database [mode: full-refresh]
2025-09-15 11:07:28 TRC select schema_name
from information_schema.schemata
where catalog_name = ( select current_catalog )
order by schema_name /* nD */ conn=conn-postgres-5j1
2025-09-15 11:07:28 TRC database col => "schema_name" native_type=NAME generic_type=string length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC query responded in 0.002491 secs
2025-09-15 11:07:28 TRC query returned 28 rows
2025-09-15 11:07:28 DBG drop table if exists "moex"."iap_quotes_imoex_tmp"
2025-09-15 11:07:28 DBG table "moex"."iap_quotes_imoex_tmp" dropped
2025-09-15 11:07:28 TRC try pausing ds.1757923648112.slb
2025-09-15 11:07:28 TRC SELECT
  pg_attribute.attname AS column_name,
  pg_catalog.format_type(pg_attribute.atttypid, pg_attribute.atttypmod) as data_type,
  CASE pg_attribute.atttypid
    WHEN 21 /*int2*/ THEN 16
    WHEN 23 /*int4*/ THEN 32
    WHEN 20 /*int8*/ THEN 64
    WHEN 1700 /*numeric*/ THEN
        CASE WHEN pg_attribute.atttypmod = -1
              THEN null
              ELSE ((pg_attribute.atttypmod - 4) >> 16) & 65535     -- calculate the precision
              END
    WHEN 700 /*float4*/ THEN 24 /*FLT_MANT_DIG*/
    WHEN 701 /*float8*/ THEN 53 /*DBL_MANT_DIG*/
    ELSE null
  END AS precision,
  CASE
    WHEN pg_attribute.atttypid IN (21, 23, 20) THEN 0
    WHEN pg_attribute.atttypid IN (1700) THEN
        CASE
            WHEN pg_attribute.atttypmod = -1 THEN null
            ELSE (pg_attribute.atttypmod - 4) & 65535            -- calculate the scale
        END
      ELSE null
  END AS scale
from pg_catalog.pg_class
INNER JOIN pg_catalog.pg_namespace ON pg_class.relnamespace = pg_namespace.oid
INNER JOIN pg_catalog.pg_attribute ON pg_class.oid = pg_attribute.attrelid
where 1=1
  and pg_class.relkind in ('r', 'v', 'm', 'f', 'p')
  and pg_namespace.nspname = 'moex'
  and pg_class.relname = 'iap_quotes_imoex_tmp'
  and pg_attribute.attnum >= 1
  and not pg_attribute.attisdropped
ORDER BY pg_attribute.attnum /* nD */ conn=conn-postgres-5j1
2025-09-15 11:07:28 TRC database col => "column_name" native_type=NAME generic_type=string length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "data_type" native_type=TEXT generic_type=text length=2147483647 precision=2147483647 scale=0 sourced=true 
2025-09-15 11:07:28 TRC database col => "precision" native_type=INT4 generic_type=integer length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "scale" native_type=INT4 generic_type=integer length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC query responded in 0.002670 secs
2025-09-15 11:07:28 TRC query returned 0 rows
2025-09-15 11:07:28 TRC select schema_name
from information_schema.schemata
where catalog_name = ( select current_catalog )
order by schema_name /* nD */ conn=conn-postgres-5j1
2025-09-15 11:07:28 TRC database col => "schema_name" native_type=NAME generic_type=string length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC query responded in 0.002065 secs
2025-09-15 11:07:28 TRC query returned 28 rows
2025-09-15 11:07:28 TRC ticker - string {"min_len":5,"max_len":5,"min":0,"max":-9223372036854774808,"null_cnt":0,"string_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC date - datetime {"min_len":19,"max_len":19,"min":0,"max":-9223372036854774808,"null_cnt":0,"date_cnt":2,"datetime_cnt":898,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 TRC close - decimal {"min_len":4,"max_len":7,"max_dec_len":2,"min":2794,"max":2889,"null_cnt":0,"dec_cnt":900,"total_cnt":900,"uniq_cnt":0,"checksum":0}
2025-09-15 11:07:28 DBG create table if not exists "moex"."iap_quotes_imoex_tmp" ("ticker" varchar(65500),
  "date" timestamp,
  "close" numeric)
2025-09-15 11:07:28 TRC RowsAffected: 0
2025-09-15 11:07:28 INF created table "moex"."iap_quotes_imoex_tmp"
2025-09-15 11:07:28 TRC begin
2025-09-15 11:07:28 TRC unpausing ds.1757923648112.slb
2025-09-15 11:07:28 INF streaming data
2025-09-15 11:07:28 TRC SELECT
  pg_attribute.attname AS column_name,
  pg_catalog.format_type(pg_attribute.atttypid, pg_attribute.atttypmod) as data_type,
  CASE pg_attribute.atttypid
    WHEN 21 /*int2*/ THEN 16
    WHEN 23 /*int4*/ THEN 32
    WHEN 20 /*int8*/ THEN 64
    WHEN 1700 /*numeric*/ THEN
        CASE WHEN pg_attribute.atttypmod = -1
              THEN null
              ELSE ((pg_attribute.atttypmod - 4) >> 16) & 65535     -- calculate the precision
              END
    WHEN 700 /*float4*/ THEN 24 /*FLT_MANT_DIG*/
    WHEN 701 /*float8*/ THEN 53 /*DBL_MANT_DIG*/
    ELSE null
  END AS precision,
  CASE
    WHEN pg_attribute.atttypid IN (21, 23, 20) THEN 0
    WHEN pg_attribute.atttypid IN (1700) THEN
        CASE
            WHEN pg_attribute.atttypmod = -1 THEN null
            ELSE (pg_attribute.atttypmod - 4) & 65535            -- calculate the scale
        END
      ELSE null
  END AS scale
from pg_catalog.pg_class
INNER JOIN pg_catalog.pg_namespace ON pg_class.relnamespace = pg_namespace.oid
INNER JOIN pg_catalog.pg_attribute ON pg_class.oid = pg_attribute.attrelid
where 1=1
  and pg_class.relkind in ('r', 'v', 'm', 'f', 'p')
  and pg_namespace.nspname = 'moex'
  and pg_class.relname = 'iap_quotes_imoex_tmp'
  and pg_attribute.attnum >= 1
  and not pg_attribute.attisdropped
ORDER BY pg_attribute.attnum /* nD */ conn=conn-postgres-5j1
2025-09-15 11:07:28 TRC database col => "column_name" native_type=NAME generic_type=string length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "data_type" native_type=TEXT generic_type=text length=2147483647 precision=2147483647 scale=0 sourced=true 
2025-09-15 11:07:28 TRC database col => "precision" native_type=INT4 generic_type=integer length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "scale" native_type=INT4 generic_type=integer length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC query responded in 0.002096 secs
2025-09-15 11:07:28 TRC query returned 3 rows
2025-09-15 11:07:28 TRC database col => "ticker" native_type=character varying(65500) generic_type=text length=0 precision=0 scale=0 sourced=true  
2025-09-15 11:07:28 TRC database col => "date" native_type=timestamp without time zone generic_type=timestamp length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "close" native_type=numeric generic_type=decimal length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC closed ds.1757923648081.72C-0
2025-09-15 11:07:28 TRC Pushed 89409 rows for ds.1757923648081.72C
2025-09-15 11:07:28 TRC closed ds.1757923648112.slb-0
2025-09-15 11:07:28 TRC Pushed 89409 rows for ds.1757923648112.slb
2025-09-15 11:07:28 TRC commiting
2025-09-15 11:07:28 TRC COPY 89409 ROWS
2025-09-15 11:07:28 TRC executing defer functions
2025-09-15 11:07:28 DBG select count(*) cnt from "moex"."iap_quotes_imoex_tmp"
2025-09-15 11:07:28 TRC database col => "cnt" native_type=INT8 generic_type=bigint length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC query responded in 0.007837 secs
2025-09-15 11:07:28 TRC new ds.Start ds.1757923648868.WSf
2025-09-15 11:07:28 TRC first row of ds.1757923648868.WSf => []interface {}{89409}
2025-09-15 11:07:28 TRC new batch ds.1757923648868.WSf-0
2025-09-15 11:07:28 TRC closed ds.1757923648868.WSf-0
2025-09-15 11:07:28 TRC Pushed 1 rows for ds.1757923648868.WSf
2025-09-15 11:07:28 TRC query returned 1 rows
2025-09-15 11:07:28 TRC ticker - string {"min_len":5,"max_len":5,"min":0,"max":0,"null_cnt":0,"string_cnt":89409,"total_cnt":89409,"uniq_cnt":0,"checksum":447045}
2025-09-15 11:07:28 TRC date - datetime {"min_len":19,"max_len":19,"min":0,"max":1756752600000000,"null_cnt":0,"date_cnt":206,"datetime_cnt":89203,"total_cnt":89409,"uniq_cnt":0,"checksum":8567999746223587072}
2025-09-15 11:07:28 TRC close - decimal {"min_len":4,"max_len":7,"max_dec_len":2,"min":0,"max":3370,"null_cnt":0,"int_cnt":933,"dec_cnt":88476,"total_cnt":89409,"uniq_cnt":0,"checksum":261022260}
2025-09-15 11:07:28 TRC begin
2025-09-15 11:07:28 DBG drop table if exists "moex"."iap_quotes_imoex"
2025-09-15 11:07:28 DBG table "moex"."iap_quotes_imoex" dropped
2025-09-15 11:07:28 TRC SELECT
  pg_attribute.attname AS column_name,
  pg_catalog.format_type(pg_attribute.atttypid, pg_attribute.atttypmod) as data_type,
  CASE pg_attribute.atttypid
    WHEN 21 /*int2*/ THEN 16
    WHEN 23 /*int4*/ THEN 32
    WHEN 20 /*int8*/ THEN 64
    WHEN 1700 /*numeric*/ THEN
        CASE WHEN pg_attribute.atttypmod = -1
              THEN null
              ELSE ((pg_attribute.atttypmod - 4) >> 16) & 65535     -- calculate the precision
              END
    WHEN 700 /*float4*/ THEN 24 /*FLT_MANT_DIG*/
    WHEN 701 /*float8*/ THEN 53 /*DBL_MANT_DIG*/
    ELSE null
  END AS precision,
  CASE
    WHEN pg_attribute.atttypid IN (21, 23, 20) THEN 0
    WHEN pg_attribute.atttypid IN (1700) THEN
        CASE
            WHEN pg_attribute.atttypmod = -1 THEN null
            ELSE (pg_attribute.atttypmod - 4) & 65535            -- calculate the scale
        END
      ELSE null
  END AS scale
from pg_catalog.pg_class
INNER JOIN pg_catalog.pg_namespace ON pg_class.relnamespace = pg_namespace.oid
INNER JOIN pg_catalog.pg_attribute ON pg_class.oid = pg_attribute.attrelid
where 1=1
  and pg_class.relkind in ('r', 'v', 'm', 'f', 'p')
  and pg_namespace.nspname = 'moex'
  and pg_class.relname = 'iap_quotes_imoex'
  and pg_attribute.attnum >= 1
  and not pg_attribute.attisdropped
ORDER BY pg_attribute.attnum /* nD */ conn=conn-postgres-5j1
2025-09-15 11:07:28 TRC database col => "column_name" native_type=NAME generic_type=string length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "data_type" native_type=TEXT generic_type=text length=2147483647 precision=2147483647 scale=0 sourced=true 
2025-09-15 11:07:28 TRC database col => "precision" native_type=INT4 generic_type=integer length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "scale" native_type=INT4 generic_type=integer length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC query responded in 0.001576 secs
2025-09-15 11:07:28 TRC query returned 0 rows
2025-09-15 11:07:28 TRC select schema_name
from information_schema.schemata
where catalog_name = ( select current_catalog )
order by schema_name /* nD */ conn=conn-postgres-5j1
2025-09-15 11:07:28 TRC database col => "schema_name" native_type=NAME generic_type=string length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC query responded in 0.001592 secs
2025-09-15 11:07:28 TRC query returned 28 rows
2025-09-15 11:07:28 TRC ticker - string {"min_len":5,"max_len":5,"min":0,"max":0,"null_cnt":0,"string_cnt":89409,"total_cnt":89409,"uniq_cnt":0,"checksum":447045}
2025-09-15 11:07:28 TRC date - datetime {"min_len":19,"max_len":19,"min":0,"max":1756752600000000,"null_cnt":0,"date_cnt":206,"datetime_cnt":89203,"total_cnt":89409,"uniq_cnt":0,"checksum":8567999746223587072}
2025-09-15 11:07:28 TRC close - decimal {"min_len":4,"max_len":7,"max_dec_len":2,"min":0,"max":3370,"null_cnt":0,"int_cnt":933,"dec_cnt":88476,"total_cnt":89409,"uniq_cnt":0,"checksum":261022260}
2025-09-15 11:07:28 DBG create table if not exists "moex"."iap_quotes_imoex" ("ticker" varchar(65500),
  "date" timestamp,
  "close" numeric)
2025-09-15 11:07:28 TRC RowsAffected: 0
2025-09-15 11:07:28 INF created table "moex"."iap_quotes_imoex"
2025-09-15 11:07:28 TRC SELECT
  pg_attribute.attname AS column_name,
  pg_catalog.format_type(pg_attribute.atttypid, pg_attribute.atttypmod) as data_type,
  CASE pg_attribute.atttypid
    WHEN 21 /*int2*/ THEN 16
    WHEN 23 /*int4*/ THEN 32
    WHEN 20 /*int8*/ THEN 64
    WHEN 1700 /*numeric*/ THEN
        CASE WHEN pg_attribute.atttypmod = -1
              THEN null
              ELSE ((pg_attribute.atttypmod - 4) >> 16) & 65535     -- calculate the precision
              END
    WHEN 700 /*float4*/ THEN 24 /*FLT_MANT_DIG*/
    WHEN 701 /*float8*/ THEN 53 /*DBL_MANT_DIG*/
    ELSE null
  END AS precision,
  CASE
    WHEN pg_attribute.atttypid IN (21, 23, 20) THEN 0
    WHEN pg_attribute.atttypid IN (1700) THEN
        CASE
            WHEN pg_attribute.atttypmod = -1 THEN null
            ELSE (pg_attribute.atttypmod - 4) & 65535            -- calculate the scale
        END
      ELSE null
  END AS scale
from pg_catalog.pg_class
INNER JOIN pg_catalog.pg_namespace ON pg_class.relnamespace = pg_namespace.oid
INNER JOIN pg_catalog.pg_attribute ON pg_class.oid = pg_attribute.attrelid
where 1=1
  and pg_class.relkind in ('r', 'v', 'm', 'f', 'p')
  and pg_namespace.nspname = 'moex'
  and pg_class.relname = 'iap_quotes_imoex_tmp'
  and pg_attribute.attnum >= 1
  and not pg_attribute.attisdropped
ORDER BY pg_attribute.attnum /* nD */ conn=conn-postgres-5j1
2025-09-15 11:07:28 TRC database col => "column_name" native_type=NAME generic_type=string length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "data_type" native_type=TEXT generic_type=text length=2147483647 precision=2147483647 scale=0 sourced=true 
2025-09-15 11:07:28 TRC database col => "precision" native_type=INT4 generic_type=integer length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "scale" native_type=INT4 generic_type=integer length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC query responded in 0.001573 secs
2025-09-15 11:07:28 TRC query returned 3 rows
2025-09-15 11:07:28 TRC database col => "ticker" native_type=character varying(65500) generic_type=text length=0 precision=0 scale=0 sourced=true  
2025-09-15 11:07:28 TRC database col => "date" native_type=timestamp without time zone generic_type=timestamp length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "close" native_type=numeric generic_type=decimal length=0 precision=0 scale=0 sourced=false
2025-09-15 11:07:28 TRC SELECT
  pg_attribute.attname AS column_name,
  pg_catalog.format_type(pg_attribute.atttypid, pg_attribute.atttypmod) as data_type,
  CASE pg_attribute.atttypid
    WHEN 21 /*int2*/ THEN 16
    WHEN 23 /*int4*/ THEN 32
    WHEN 20 /*int8*/ THEN 64
    WHEN 1700 /*numeric*/ THEN
        CASE WHEN pg_attribute.atttypmod = -1
              THEN null
              ELSE ((pg_attribute.atttypmod - 4) >> 16) & 65535     -- calculate the precision
              END
    WHEN 700 /*float4*/ THEN 24 /*FLT_MANT_DIG*/
    WHEN 701 /*float8*/ THEN 53 /*DBL_MANT_DIG*/
    ELSE null
  END AS precision,
  CASE
    WHEN pg_attribute.atttypid IN (21, 23, 20) THEN 0
    WHEN pg_attribute.atttypid IN (1700) THEN
        CASE
            WHEN pg_attribute.atttypmod = -1 THEN null
            ELSE (pg_attribute.atttypmod - 4) & 65535            -- calculate the scale
        END
      ELSE null
  END AS scale
from pg_catalog.pg_class
INNER JOIN pg_catalog.pg_namespace ON pg_class.relnamespace = pg_namespace.oid
INNER JOIN pg_catalog.pg_attribute ON pg_class.oid = pg_attribute.attrelid
where 1=1
  and pg_class.relkind in ('r', 'v', 'm', 'f', 'p')
  and pg_namespace.nspname = 'moex'
  and pg_class.relname = 'iap_quotes_imoex'
  and pg_attribute.attnum >= 1
  and not pg_attribute.attisdropped
ORDER BY pg_attribute.attnum /* nD */ conn=conn-postgres-5j1
2025-09-15 11:07:28 TRC database col => "column_name" native_type=NAME generic_type=string length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "data_type" native_type=TEXT generic_type=text length=2147483647 precision=2147483647 scale=0 sourced=true 
2025-09-15 11:07:28 TRC database col => "precision" native_type=INT4 generic_type=integer length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "scale" native_type=INT4 generic_type=integer length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC query responded in 0.001597 secs
2025-09-15 11:07:28 TRC query returned 3 rows
2025-09-15 11:07:28 TRC database col => "ticker" native_type=character varying(65500) generic_type=text length=0 precision=0 scale=0 sourced=true  
2025-09-15 11:07:28 TRC database col => "date" native_type=timestamp without time zone generic_type=timestamp length=0 precision=0 scale=0 sourced=true
2025-09-15 11:07:28 TRC database col => "close" native_type=numeric generic_type=decimal length=0 precision=0 scale=0 sourced=false
2025-09-15 11:07:28 TRC insert target fields: ticker, date, close
2025-09-15 11:07:28 DBG insert into "moex"."iap_quotes_imoex" ("ticker", "date", "close") select "ticker" as "ticker", "date" as "date", "close" as "close" from "moex"."iap_quotes_imoex_tmp"
2025-09-15 11:07:28 DBG inserted rows into "moex"."iap_quotes_imoex" from temp table "moex"."iap_quotes_imoex_tmp"
2025-09-15 11:07:28 TRC commiting
2025-09-15 11:07:28 INF inserted 89409 rows into "moex"."iap_quotes_imoex" in 0 secs [98,378 r/s]
2025-09-15 11:07:28 DBG drop table if exists "moex"."iap_quotes_imoex_tmp"
2025-09-15 11:07:28 DBG table "moex"."iap_quotes_imoex_tmp" dropped
2025-09-15 11:07:28 DBG closed "postgres" connection (conn-postgres-5j1)
2025-09-15 11:07:28 TRC executing defer functions
2025-09-15 11:07:28 TRC executing defer functions
2025-09-15 11:07:28 INF execution succeeded

flarco · 2025-09-24T15:00:30Z

flarco
Sep 24, 2025
Maintainer

Created #651

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

"colums" YAML key not working (or maybe there is a trick to use it properly?) #642

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

"colums" YAML key not working (or maybe there is a trick to use it properly?) #642

Uh oh!

Uh oh!

igrpon Sep 15, 2025

Replies: 1 comment

Uh oh!

flarco Sep 24, 2025 Maintainer

igrpon
Sep 15, 2025

flarco
Sep 24, 2025
Maintainer