Video summary

Андрей Бурмистров. Темная сторона тех. журнала – что может пойти не так при парсинге?

Main summary

Key takeaways

Technology

“Dark side” pitfalls when parsing 1C technical journal (tech journal logs)

The speaker (Andrey Burmistov) discusses how technical journal parsing can produce incorrect or incomplete analysis, even when the logs look trustworthy. He focuses on data/field inconsistencies, duplicate records, and format-specific parsing traps, especially relevant for building reliable scripts/ETL into systems like ClickHouse and for security/monitoring reporting.

Key milestones / format change

  • Recent milestone: 1C technical journal now supports JSON format (e.g., “E25… new format” is mentioned; previously only a text format).
  • Old situation: the text format had an unstable structure (line breaks, formatting issues), which made parsing inconvenient.
  • JSON benefit: each event is on its own line, making search and saving easier.
  • But: many structural quirks remain even in JSON—so parsing still requires care.

1) Duplicate properties (duplicate key fields inside one event)

What goes wrong

  • Some events contain the same property name twice (e.g., Client ID repeated).
  • Many parsers resolve duplicates by keeping the last occurrence (i.e., “last key value wins”).
  • This becomes dangerous when the two occurrences have different meanings/values.

Concrete examples

  • Transaction completion (SDBL / “completion of a transaction”):
    • A function-like property is duplicated.
    • If you configure a filter in the tech journal setup, you might be filtering the wrong occurrence (while the other may be ignored).

  • Database name / server context:

    • Property like Pprocess name is often used to store the 1C information base name, even if docs call it server context.

    • In dynamic update scenarios, you may see multiple values: processName, <databaseName>, <dynamicUpdateId>

    • If your pipeline assumes equality/regex behavior based on a single value, you may lose the first value due to duplicate-key handling.

Takeaway: Don’t assume that “property X == one value.” Duplicates can cause silent data loss and wrong joins/filters in downstream analytics.

2) Duplicate events (repeated records across different log views)

Authentication/security example (access logging)

  • In the tech journal, an exception (EXCP) may indicate multiple rapid failed login attempts (e.g., “four times” within a ~1 second range).
  • But the registration log shows fewer events (e.g., two).
  • Details reveal confusing context:
    • The tech journal may show warnings like “incorrect authorization,” but in some cases it corresponds to different behavior (e.g., OS authentication attempt, while the real failure was 1C auth with a missing username, etc.).
  • Because of these mismatches, security reporting like “how many login attempts occurred” can become incorrect if based solely on tech journal exception lines.

Takeaway: For security KPIs and attack detection counts, validate against ground-truth log sources and don’t rely on duplicated/misleading tech log entries.

Deadlock counting example

  • Two exceptions may be recorded in the tech journal for one deadlock.
  • Monitoring that counts deadlocks by substring matching (e.g., “lock conflict/deadlock”) can report inflated deadlock numbers.

Mitigation mentioned: Use additional filtering by a more specific field (e.g., exception / database exception) instead of counting plain text fragments.

3) “Ghost fields” (fields appear/disappear depending on event context)

Some properties only exist when a condition is true:

  • Example: event for setting a controlled lock has a property like escalation:
    • If escalation is false, the property does not exist in the log line.
    • If true, it appears with the full value.
  • Another field (weight connections) is described as always present, regardless of state.

Takeaway: Your schema must allow optional fields; don’t assume columns always exist.

4) Inconsistent field formats (numbers vs empty vs “cash”)

Fields like DBMS connection identifiers can vary:

  • A “connection number with DBMS/UBD” field:
    • Usually numeric, but sometimes can be empty or non-numeric (the speaker mentions “cash” as an auto-text error).
  • If your parsing template expects strictly numeric values, some lines may be skipped/unprocessed.

Other variability:

  • Session ID fields may include extra bracketed content (e.g., parent session ID in brackets).
  • Regex/filter logic that assumes a clean session ID can miss data.
  • Username fields can be named differently across events (e.g., USR vs username vs computer name / client computer name).

5) Analysis of waits/locks requires cross-object correlation

A logical limitation in typical “controlled waits” analysis:

  • An expectation event may show locks on multiple objects (e.g., accumulation register + information register).
  • To determine the specific object the wait is on, you must:
    1. find the culprit connection (via weight connections / connection number),
    2. then check which blocks were set on all involved tables/fields that share the same connection number,
    3. compute the intersection.

Takeaway: A single expectation record may not be sufficient; you need join/correlation across events.

6) Context/call-stack reconstruction is non-trivial

Context can be stored in separate events

  • Usually, an event has a context property with a call stack.
  • But sometimes the event itself lacks context, and context appears as a separate event with the same linking key (client id / event id).
  • Scripts must detect and merge those events.

Context can be split

  • Call stack may be divided:
    • part embedded in the event,
    • the rest in a separate “context event”.
  • You must “glue” them to reconstruct the full call stack.

7) Unexpected “everywhere” event type: Exception event (“böiuks transform”)

Even if your filter targets only certain events (deadlocks, SQL requests, etc.), exception events may still appear.

  • Your parser must not crash when encountering Exception, because it has a different field set.

8) Text format parsing edge cases: commas and quotes inside values

Old text journal format

  • Values are usually separated as key=value with commas.
  • If a value contains a comma or a line break, it is enclosed in apostrophes.
  • If apostrophes appear inside the value, the rules become ambiguous and require heuristics.
  • Special case described:
    • If both apostrophes and quotation marks occur, 1C chooses apostrophes as framing when counts match, and apostrophes inside are doubled.
  • Result: a “universal parser” for all old-format cases is hard.

JSON format

  • Values are always framed in quotation marks.
  • Internal quotes are escaped; quote duplication rules make it simpler/safer for parsing.

AI-assisted log analysis approaches (as suggested by the speaker)

  1. Send a log fragment + instructions to an LLM
    • Ask for grouping (e.g., by last line of context).
    • Works best when the pattern is structured and limited (1–2 event types).
    • Limited by input size (tens of MB max due to token cost).
  2. Prompt to generate a script that implements the required parsing/analysis
    • The script may “glitch” a couple of times, but can be rerun on full datasets (gigabytes) locally.

Mentioned tool

  • Google NotebookLM used as a personal knowledge base:
    • upload 1C docs, YouTube video references, logs, etc.
    • ask questions and use it as an advisor for 1C-related topics.

Main speakers / sources

  • Andrey Burmistov — main presenter (1C performance and tech journal parsing pitfalls).
  • Audience mentions a Grigory Shatrov upcoming report (not covered in detail in these subtitles).

Original video