关于谷歌云数据流：如何使用 Apache Beam 在 Python 中将有界 pcollection 转换为无界？

How to transform bounded pcollection to unbounded in Python with Apache Beam?

我正在尝试在不使用太多内存的情况下转换存储在 GCS 中的几 TB 邮件日志。

按照指南中的建议，我为每个元素添加时间戳，将其拆分为滑动窗口，并在将其发送到 GroupByKey 和之后的 ParDo 解析器之前指定了一个(聚合)触发器。这应该可以，但仍然 GroupByKey 等待所有数据到达。为什么？

我也尝试过使用 Direct 和 Google Dataflow runner。

我错过了什么？

这是代码的要点：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

known_args, pipeline_options = parse_args(sys.argv)
pipeline = beam.Pipeline(options=pipeline_options)
lines = pipeline | 'read' >> beam.io.ReadFromText(known_args.input)
parsed_json = lines \\
| 'cut' >> beam.Map(lambda x: x[39:])\\
| 'jsonify' >> beam.Map(json_loads).with_outputs('invalid_input', main='main')

keyed_lines = parsed_json['main']\\
| 'key_filter' >> beam.Filter(lambda x: u'key' in x)\\
| 'keyer' >> beam.Map(lambda x: (x['key'], x))\\
| 'log_processed_rows' >> beam.Map(log_processed_rows)

window_trigger = trigger.DefaultTrigger()

windowed_lines = keyed_lines\\
| 'timestamp' >> beam.ParDo(AddTimestampDoFn())\\
| 'window' >> beam.WindowInto(beam.window.SlidingWindows(size=3600+600, period=3600), trigger=window_trigger,
accumulation_mode=trigger.AccumulationMode.DISCARDING) # 1 hour steps with 10 minutes of overlap

results = windowed_lines \\
| 'groupbykey' >> beam.GroupByKey()\\
| 'parse' >> beam.ParDo(ParseSendingsDoFn()).with_outputs('too_few_rows', 'invalid_rows', 'missing_recipients', main='main_valid')

output = results['main_valid'] \\
| 'format' >> beam.Map(output_format)\\
| 'write' >> beam.io.WriteToText(known_args.output, file_name_suffix=".gz")