ProcessingJobで使用可能なジョブ数を上げる方法

Last updated at 2024-09-26Posted at 2024-09-24

はじめに

こんにちは。弥生R&D室のsiidaです。R&D室ではSageMakerを使用して機械学習 (ML) のプロジェクトを進めています。SageMakerはMLのための様々な機能が搭載されたサービスであり、データ分析からモデル訓練、ひいてはワークフローの構築まで、SageMakerの中で完結させることができます。

SageMakerにはProcessingJobという機能があり、こちらはコマンドをジョブの形でサーバ上で実行できるというものです。重い計算を実行したり、再現性のある実験を行うために有用な機能ですが、同時に実行できるジョブの数には制限があります。今回は、この使用可能なジョブ数を上げる方法を紹介します。

背景

ProcessingJobでジョブを作成しようとしたところ、ジョブ多すぎ！という感じのエラーが。。。

なお、ジョブの作成には、下記のPythonラッパーを使用しました。

発生したエラー

ERROR:sagemaker:Please check the troubleshooting guide for common errors: https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-python-sdk-troubleshooting.html#sagemaker-python-sdk-troubleshooting-create-processing-job
Traceback (most recent call last):
  File "/home/sagemaker-user/processing-job-sample/trigger.py", line 116, in <module>
    main()
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/sagemaker-user/processing-job-sample/trigger.py", line 105, in main
    run_processor(
  File "/home/sagemaker-user/processing-job-sample/src/processing_job_sample/processing.py", line 109, in run_processor
    processor.run(
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/sagemaker/workflow/pipeline_context.py", line 346, in wrapper
    return run_func(*args, **kwargs)
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/sagemaker/processing.py", line 1781, in run
    return super().run(
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/sagemaker/workflow/pipeline_context.py", line 346, in wrapper
    return run_func(*args, **kwargs)
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/sagemaker/processing.py", line 681, in run
    self.latest_job = ProcessingJob.start_new(
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/sagemaker/processing.py", line 917, in start_new
    processor.sagemaker_session.process(**process_args)
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/sagemaker/session.py", line 1589, in process
    self._intercept_create_request(process_request, submit, self.process.__name__)
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/sagemaker/session.py", line 6606, in _intercept_create_request
    return create(request)
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/sagemaker/session.py", line 1587, in submit
    raise e
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/sagemaker/session.py", line 1577, in submit
    self.sagemaker_client.create_processing_job(**request)
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/sagemaker-user/.cache/pypoetry/virtualenvs/processing-job-sample-KaeK58QC-py3.10/lib/python3.10/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateProcessingJob operation: The account-level service limit 'ml.m5.24xlarge for processing job usage' is 10 Instances. Please use AWS Service Quotas to request an increase for this quota. If AWS Service Quotas is not available, contact AWS support to request an increase for this quota.

現在の設定では、使用しているインスタンスでは同時に10件のジョブまでしか利用できないとのことでした。

この件数を引き上げたいと思います。