[[["容易理解","easyToUnderstand","thumb-up"],["確實解決了我的問題","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["難以理解","hardToUnderstand","thumb-down"],["資訊或程式碼範例有誤","incorrectInformationOrSampleCode","thumb-down"],["缺少我需要的資訊/範例","missingTheInformationSamplesINeed","thumb-down"],["翻譯問題","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["上次更新時間：2025-08-18 (世界標準時間)。"],[[["\u003cp\u003eCloud Profiler is a tool that continuously gathers CPU and memory usage data from production applications, and integrates with Dataflow to help identify resource-intensive parts of pipelines.\u003c/p\u003e\n"],["\u003cp\u003eTo enable Cloud Profiler for Dataflow pipelines, you must use Apache Beam SDK version 2.33.0 or later, and start the pipeline with specific options like \u003ccode\u003e--dataflowServiceOptions=enable_google_cloud_profiler\u003c/code\u003e for CPU profiling.\u003c/p\u003e\n"],["\u003cp\u003eYour project needs sufficient Cloud Profiler quota, and the Dataflow worker service account requires appropriate permissions, such as \u003ccode\u003ecloudprofiler.profiles.create\u003c/code\u003e, to create profiles.\u003c/p\u003e\n"],["\u003cp\u003eThe Profiler page displays a flame graph that allows you to visualize the execution time of code frames, while the use of marker frames, like those with the suffix \u003ccode\u003einvokeProcessElement\u003c/code\u003e help to pinpoint user code.\u003c/p\u003e\n"],["\u003cp\u003eIf profiling data is not displayed, check the Apache Beam SDK version, ensure the project has enough quota, verify the job ran long enough, confirm the correct job is selected, and review the service name.\u003c/p\u003e\n"]]],[],null,["Cloud Profiler is a statistical, low-overhead profiler that continuously\ngathers CPU usage and memory allocation information from your production\napplications. For more details, see [Profiling concepts](/profiler/docs/concepts-profiling).\nTo troubleshoot or monitor pipeline performance, use Dataflow\nintegration with Cloud Profiler to identify the parts of the pipeline code\nconsuming the most resources.\n\nFor troubleshooting tips and debugging strategies for building or running your\nDataflow pipeline, see\n[Troubleshooting and debugging pipelines](/dataflow/docs/guides/troubleshooting-your-pipeline).\n\nBefore you begin\n\nUnderstand [profiling concepts](/profiler/docs/concepts-profiling) and\nfamiliarize yourself with the Profiler interface.\nFor information about how to get started with the Profiler\ninterface, see\n[Select the profiles to analyze](/profiler/docs/selecting-profiles).\n\nThe Cloud Profiler API must be enabled for your project before your job is started.\nIt is enabled automatically the first time you [visit the Profiler\npage](/profiler/docs/measure-app-performance#before_you_begin).\nAlternatively, you can enable the Cloud Profiler API by using the\n[Google Cloud CLI](/profiler/docs/profiling-python#enabling-profiler) `gcloud` command-line tool or the Google Cloud console.\n\nTo use Cloud Profiler, your project must have enough [quota](/profiler/quotas).\nIn addition, the\n[worker service account](/dataflow/docs/concepts/security-and-permissions#worker-service-account)\nfor the Dataflow job must have\nappropriate permissions for Profiler. For example, to create\nprofiles, the worker service account must have the `cloudprofiler.profiles.create`\npermission, which is included in the Cloud Profiler Agent\n(`roles/cloudprofiler.agent`) IAM role.\nFor more information, see [Access control with IAM](/profiler/docs/iam).\n\nEnable Cloud Profiler for Dataflow pipelines\n\nCloud Profiler is available for Dataflow pipelines written in\nApache Beam SDK for Java and Python, version 2.33.0 or later. Python\npipelines must use Dataflow Runner v2. Cloud Profiler can be\nenabled at pipeline start time. The amortized CPU and memory overhead is\nexpected to be less than 1% for your pipelines. \n\nJava\n\n\nTo enable CPU profiling, start the pipeline with the following option.\n\n\n`--dataflowServiceOptions=enable_google_cloud_profiler`\n\nTo enable heap profiling, start the pipeline with the following options. Heap profiling\nrequires Java 11 or higher.\n\n`--dataflowServiceOptions=enable_google_cloud_profiler`\n\n`--dataflowServiceOptions=enable_google_cloud_heap_sampling`\n| Note: The pipeline option `--dataflowServiceOptions` is the\n| Dataflow preferred way to enable Dataflow\n| features. Alternatively, you can use `--experiments`.\n\nPython\n\nTo use Cloud Profiler, your Python pipeline must run with Dataflow\n[Runner v2](/dataflow/docs/runner-v2).\n\nTo enable CPU profiling, start the pipeline with the following option. Heap profiling is not yet supported for Python.\n\n`--dataflow_service_options=enable_google_cloud_profiler`\n| Note: The pipeline option `--dataflow_service_options` is\n| the Dataflow preferred way to enable Dataflow\n| features. Alternatively, you can use `--experiments`.\n\nGo\n\nTo enable CPU and heap profiling, start the pipeline with the following option.\n\n`--dataflow_service_options=enable_google_cloud_profiler`\n| Note: The pipeline option `--dataflow_service_options` is\n| the Dataflow preferred way to enable Dataflow\n| features. Alternatively, you can use `--experiments`.\n\nIf you deploy your pipelines from [Dataflow templates](/dataflow/docs/concepts/dataflow-templates) and want to enable Cloud Profiler,\nspecify the `enable_google_cloud_profiler` and\n`enable_google_cloud_heap_sampling` flags as additional experiments. \n\nConsole\n\nIf you use a [Google-provided\ntemplate](/dataflow/docs/guides/templates/provided-templates), you can specify the flags on the Dataflow **Create\njob from template** page in the **Additional experiments** field.\n\ngcloud\n\nIf you use the Google Cloud CLI to run\ntemplates, either [`gcloud\ndataflow jobs run`](/sdk/gcloud/reference/dataflow/jobs/run) or [`gcloud dataflow flex-template run`](/sdk/gcloud/reference/dataflow/flex-\ntemplate/run), depending on\nthe template type, use the `--additional-experiments`\noption to specify the flags.\n\nAPI\n\nIf you use the REST\nAPI to run templates, depending on the template type, specify the flags using the\n`additionalExperiments` field of the runtime environment, either [`RuntimeEnvironment`](/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment) or [`FlexTemplateRuntimeEnvironment`](/dataflow/docs/reference/rest/v1b3/projects.locations.flexTemplates/launch\n#FlexTemplateRuntimeEnvironment).\n\nView the profiling data\n\nIf Cloud Profiler is enabled, a link to the Profiler page is\nshown on the job page.\n\nOn the Profiler page, you can also find the profiling data for\nyour Dataflow pipeline. The **Service** is your job name and the\n**Version** is your job ID.\n\nUsing the Cloud Profiler\n\nThe Profiler page contains a [flame graph](/profiler/docs/concepts-flame) which displays statistics for each frame running on a worker.\nIn the horizontal direction, you can see how long each frame took to execute in terms of CPU time.\nIn the vertical direction, you can see stack traces and code running in parallel.\nThe stack traces are dominated by runner infrastructure code. For debugging purposes we are usually interested in user code execution, which is typically found near the bottom tips of the graph.\nUser code can be identified by looking for **marker frames** , which represent runner code that is known to only call into user code.\nIn the case of the Beam ParDo runner, a dynamic adapter layer is created to invoke the user-supplied DoFn method signature.\nThis layer can be identified as a frame with the **invokeProcessElement** suffix.\nThe following image shows a demonstration of finding a **marker frame**.\n\nAfter clicking on an interesting marker frame, the flame graph focuses on that stack trace, giving a good sense of long running user code.\nThe slowest operations can indicate where bottlenecks have formed and present opportunities for optimization.\nIn the following example, it is possible to see that Global Windowing is being used with a ByteArrayCoder.\nIn this case, the coder might be a good area for optimization because it is taking up significant CPU time compared to the ArrayList and HashMap operations.\n| **Note:** The following filter can be applied to quickly find and isolate all marker frames: `Show from frame: invokeProcessElement`.\n\nTroubleshoot Cloud Profiler\n\nIf you enable Cloud Profiler and your pipeline doesn't generate profiling\ndata, one of the following conditions might be the cause.\n\n- Your pipeline uses an older Apache Beam SDK version. To use\n Cloud Profiler, you need to use Apache Beam SDK version 2.33.0 or later.\n You can view your pipeline's Apache Beam SDK version on the job page. If\n your job is created from Dataflow templates, the templates must\n use the [supported SDK versions](/dataflow/docs/support/sdk-version-support-status).\n\n- Your project is running out of Cloud Profiler quota. You can view the\n quota usage from your project's quota page. An error such as `Failed to\n collect and upload profile whose profile type is WALL` can occur if the\n Cloud Profiler quota is exceeded. The Cloud Profiler service\n rejects the profiling data if you've reached your quota. For more information about Cloud Profiler quotas, see [Quotas and Limits](/profiler/quotas).\n\n- Your job didn't run long enough to generate data for Cloud Profiler.\n Jobs that run for short durations, such as less than five minutes, might not\n provide enough profiling data for Cloud Profiler to generate results.\n\nThe Cloud Profiler agent is installed during Dataflow worker\nstartup. Log messages generated by Cloud Profiler are available in the log\ntype `dataflow.googleapis.com/worker-startup`.\n\nSometimes, profiling data exists but Cloud Profiler does not display any\noutput. The Profiler displays a message similar to, `There were\nprofiles collected for the specified time range, but none match the current\nfilters`.\n\nTo resolve this issue, try the following troubleshooting steps.\n\n- Make sure that the timespan and end time in the Profiler are inclusive of the job's elapsed time.\n\n- Confirm that the correct job is selected in the Profiler. The **Service** is your job name.\n\n- Confirm that the `job_name` pipeline option has the same value as the job name on the Dataflow job page.\n\n- If you specified a [service-name argument](/profiler/docs/profiling-python#svc-name-and-version) when you loaded the Profiler agent, confirm that the service name is configured correctly."]]
