appcfg.py doesn't exist

[jdwall]

Hi,

I deployed a DO Droplet to give this a shot. I was able to get through all of the installation, until, the step where you are to interact with appcfg.py which doesn't exist. I feel this should be obvious, but, I am totally lost as to where it could be.

root@python-2gb-nyc3-01:/home/PerfKitExplorer# bash compile.sh
** Compilation initializing.
* Clean out the existing deployment content.
[01:04:15] Using gulpfile /home/PerfKitExplorer/gulpfile.js
[01:04:15] Starting 'third_party'...
[01:04:15] Finished 'third_party' after 38 ms
[01:04:15] Starting 'common'...
[01:04:16] Finished 'common' after 11 ms
[01:04:16] Starting 'prod'...
[01:04:16] Finished 'prod' after 11 ms
[01:04:16] Starting 'default'...
[01:04:16] Finished 'default' after 12 μs

** Compilation complete.
root@python-2gb-nyc3-01:/home/PerfKitExplorer# appcfg.py --oauth2 update deploy
appcfg.py: command not found

Even in /home/PerfKitExplorer/deploy/ there isn't a appcfg.py. Using find I couldn't find that file either. Did I mess up something which would have created appcfg.py or is the documentation need to be updated? Pulled from master and am up to commit 504ab84.

root@python-2gb-nyc3-01:~# gcloud -v
Google Cloud SDK 136.0.0
bq 2.0.24
bq-nix 2.0.24
core 2016.11.30
core-nix 2016.11.07
gcloud
gsutil 4.22
gsutil-nix 4.18

root@python-2gb-nyc3-01:~# python --version
Python 2.7.12

root@python-2gb-nyc3-01:~# java -version
java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-3)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)

root@python-2gb-nyc3-01:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.1 LTS
Release:	16.04
Codename:	xenial

[tedsta]

Have you installed the App Engine SDK for Python? If so, you may need to create symlinks in /usr/bin or similar, or modify your path. See here

[jdwall]

I did install the App Engine SDK for Python. I destroyed the Droplet in the time since. I will redo the steps again this afternoon or tomorrow and report back my findings. Thanks in advance, @tedsta

[jdwall]

Hi @tedsta! I realized now, in retrospect, that I failed to provide the $PATH so the appcfg.py piece is fixed. However, now I am finding myself that I can't deploy the app to App Engine as there are limits of 10,000 files for new deployments? It's a bummer the limits employed disallow me from deploying until I send in a Google form. :(

2016-12-06 21:40:16,787 DEBUG    root            (gcloud.app.deploy) INVALID_ARGUMENT: This deployment has too many files. New versions are limited to 10000 files for this app.
Traceback (most recent call last):
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 740, in Execute
    resources = args.calliope_command.Run(cli=self, args=args)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 1678, in Run
    resources = command_instance.Run(args)
  File "/usr/lib64/google-cloud-sdk/lib/surface/app/deploy.py", line 53, in Run
    return deploy_util.RunDeploy(args, app_create=True)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 418, in RunDeploy
    all_services)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 257, in Deploy
    endpoints_info)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 114, in DeployService
    self.client.apps_services_versions.Create, create_request)
  File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/api/requests.py", line 67, in MakeRequest
    raise err() if err else exc
HttpException: INVALID_ARGUMENT: This deployment has too many files. New versions are limited to 10000 files for this app.
2016-12-06 21:40:16,788 ERROR    root            (gcloud.app.deploy) INVALID_ARGUMENT: This deployment has too many files. New versions are limited to 10000 files for this app.

I wasn't able to get appcfg.py to deploy the app, so I used gcloud app deploy. I had to comment out application and version for it to go through. I am not sure if that affects anything here, but, wanted to provide you context just in case it does.

[root@centos-2gb-nyc3-01 PerfKitExplorer]# appcfg.py --oauth2 update deploy --noauth_local_webserver

Note: the --oauth2 flag is now the default and can be omitted.

09:46 PM Application: REDACTED-perfkit-explorer; version: beta
09:46 PM Host: appengine.google.com
09:46 PM Starting update of app: REDACTED-perfkit-explorer, version: beta
09:46 PM Getting current resource limits.
09:46 PM Scanning files on local disk.
Could not guess mimetype for client/third_party/ui-grid/ui-grid.eot.  Using application/octet-stream.
Could not guess mimetype for client/third_party/ui-grid/ui-grid.svg.  Using application/octet-stream.
Could not guess mimetype for client/third_party/jquery/jquery.min.map.  Using application/octet-stream.
Could not guess mimetype for client/third_party/bootstrap-ui/fonts/glyphicons-halflings-regular.eot.  Using application/octet-stream.
Could not guess mimetype for client/third_party/bootstrap-ui/fonts/glyphicons-halflings-regular.svg.  Using application/octet-stream.
Could not guess mimetype for client/third_party/angular-material/angular-material.scss.  Using application/octet-stream.
Could not guess mimetype for client/third_party/angular/angular-animate.min.js.map.  Using application/octet-stream.
Could not guess mimetype for client/third_party/angular/angular-aria.min.js.map.  Using application/octet-stream.
Could not guess mimetype for client/third_party/angular/angular-sanitize.min.js.map.  Using application/octet-stream.
Could not guess mimetype for client/third_party/angular/angular.min.js.gzip.  Using application/octet-stream.
Could not guess mimetype for client/third_party/angular/angular.min.js.map.  Using application/octet-stream.
Could not guess mimetype for client/components/widget/data_viz/gviz/gviz-charts.json.  Using application/octet-stream.
Error 404: --- begin server output ---
This application does not exist (project_id=u'REDACTED-perfkit-explorer'). To create an App Engine application in this project, run "gcloud beta app create" in your console.
--- end server output ---
[root@centos-2gb-nyc3-01 PerfKitExplorer]#

[tedsta]

That is a bummer. I've pinged the original developer internally and will post back here with any updates.

[jmuharsky]

It looks like venv (virtual environment for testing) is getting copied into the deploy folder. If you remove the deploy/server/venv folder before deployment, you should be well under the 10k file limit. I'll look at fixing this step shortly.

[jdwall]

@jmuharsky I am not seeing a folder called venv. Any idea on where it might be hiding?

[root@centos-2gb-nyc3-01 server]# pwd
/root/projects/PerfKitExplorer/deploy/server
[root@centos-2gb-nyc3-01 deploy]# tree /root/projects/PerfKitExplorer/deploy/
/root/projects/PerfKitExplorer/deploy/
├── appengine_config.py
├── app.yaml
├── client
│   ├── components
│   │   └── widget
│   │       └── data_viz
│   │           └── gviz
│   │               └── gviz-charts.json
│   ├── perfkit_scripts.js
│   ├── perfkit_styles.css
│   ├── perfkit_templates.js
│   └── third_party
│       ├── angular
│       │   ├── angular-animate.js
│       │   ├── angular-animate.min.js
│       │   ├── angular-animate.min.js.map
│       │   ├── angular-aria.js
│       │   ├── angular-aria.min.js
│       │   ├── angular-aria.min.js.map
│       │   ├── angular.js
│       │   ├── angular.min.js
│       │   ├── angular.min.js.gzip
│       │   ├── angular.min.js.map
│       │   ├── angular-mocks.js
│       │   ├── angular-sanitize.js
│       │   ├── angular-sanitize.min.js
│       │   └── angular-sanitize.min.js.map
│       ├── angular-material
│       │   ├── angular-material.css
│       │   ├── angular-material.js
│       │   ├── angular-material.min.css
│       │   ├── angular-material.min.js
│       │   └── angular-material.scss
│       ├── angular-ui-router
│       │   ├── angular-ui-router.js
│       │   └── angular-ui-router.min.js
│       ├── bootstrap-ui
│       │   ├── css
│       │   │   └── bootstrap.min.css
│       │   ├── fonts
│       │   │   ├── glyphicons-halflings-regular.eot
│       │   │   ├── glyphicons-halflings-regular.svg
│       │   │   ├── glyphicons-halflings-regular.ttf
│       │   │   ├── glyphicons-halflings-regular.woff
│       │   │   └── glyphicons-halflings-regular.woff2
│       │   ├── ui-bootstrap-tpls.js
│       │   └── ui-bootstrap-tpls.min.js
│       ├── codemirror
│       │   ├── codemirror.css
│       │   ├── codemirror.js
│       │   └── mode
│       │       ├── javascript
│       │       │   └── javascript.js
│       │       └── sql
│       │           └── sql.js
│       ├── jquery
│       │   ├── jquery.min.js
│       │   └── jquery.min.map
│       ├── showdown
│       │   ├── ng-showdown.js
│       │   ├── ng-showdown.min.js
│       │   ├── showdown.js
│       │   └── showdown.min.js
│       └── ui-grid
│           ├── ui-grid.css
│           ├── ui-grid.eot
│           ├── ui-grid.js
│           ├── ui-grid.min.css
│           ├── ui-grid.min.js
│           ├── ui-grid.svg
│           ├── ui-grid.ttf
│           └── ui-grid.woff
├── config
│   ├── big_query_v2_rest.json
│   ├── credentials.json
│   ├── data_source_config.json
│   └── data_source_config_mock.json
├── index.yaml
├── __init__.py
└── server
    ├── perfkit
    │   ├── common
    │   │   ├── big_query_client.py
    │   │   ├── big_query_client_test.py
    │   │   ├── big_query_result_pivot.py
    │   │   ├── big_query_result_pivot_test.py
    │   │   ├── big_query_result_util.py
    │   │   ├── big_query_result_util_test.py
    │   │   ├── credentials_lib.py
    │   │   ├── credentials_lib_test.py
    │   │   ├── data_source_config.py
    │   │   ├── data_source_config_test.py
    │   │   ├── datetime_util.py
    │   │   ├── datetime_util_test.py
    │   │   ├── gae_big_query_client.py
    │   │   ├── gae_cloud_sql_client.py
    │   │   ├── gae_test_util.py
    │   │   ├── http_util.py
    │   │   ├── http_util_test.py
    │   │   ├── __init__.py
    │   │   └── mock_big_query_client.py
    │   ├── explorer
    │   │   ├── handlers
    │   │   │   ├── base.py
    │   │   │   ├── dashboard.py
    │   │   │   ├── dashboard_test.py
    │   │   │   ├── data.py
    │   │   │   ├── data_test.py
    │   │   │   ├── explorer_config.py
    │   │   │   ├── explorer_config_test.py
    │   │   │   ├── __init__.py
    │   │   │   ├── pages.py
    │   │   │   ├── pages_test.py
    │   │   │   └── templates
    │   │   │       ├── compare.html
    │   │   │       ├── dashboard-admin.html
    │   │   │       ├── explorer.html
    │   │   │       ├── page-base.html
    │   │   │       └── third-party-scripts.html
    │   │   ├── __init__.py
    │   │   ├── model
    │   │   │   ├── dashboard_fields.py
    │   │   │   ├── dashboard.py
    │   │   │   ├── dashboard_test.py
    │   │   │   ├── error_fields.py
    │   │   │   ├── explorer_config.py
    │   │   │   ├── explorer_config_test.py
    │   │   │   ├── __init__.py
    │   │   │   └── settings.py
    │   │   ├── samples_mart
    │   │   │   ├── explorer_method.py
    │   │   │   ├── explorer_method_test.py
    │   │   │   ├── __init__.py
    │   │   │   ├── label_manager.py
    │   │   │   ├── label_manager_test.py
    │   │   │   ├── product_labels.py
    │   │   │   └── product_labels_test.py
    │   │   └── util
    │   │       ├── explorer_config_util.py
    │   │       ├── explorer_config_util_test.py
    │   │       ├── __init__.py
    │   │       └── user_validator.py
    │   ├── ext
    │   │   ├── cloudsql
    │   │   │   ├── handlers
    │   │   │   │   ├── cloudsql_config.py
    │   │   │   │   ├── cloudsql_config_test.py
    │   │   │   │   └── __init__.py
    │   │   │   ├── __init__.py
    │   │   │   └── models
    │   │   │       ├── cloudsql_config.py
    │   │   │       ├── cloudsql_config_test.py
    │   │   │       └── __init__.py
    │   │   └── __init__.py
    │   ├── __init__.py
    │   └── test_util.py
    └── third_party
        ├── apiclient
        │   ├── channel.py
        │   ├── discovery.py
        │   ├── errors.py
        │   ├── http.py
        │   ├── __init__.py
        │   ├── mimeparse.py
        │   ├── model.py
        │   ├── sample_tools.py
        │   └── schema.py
        ├── dateutil
        │   ├── easter.py
        │   ├── __init__.py
        │   ├── parser.py
        │   ├── relativedelta.py
        │   ├── rrule.py
        │   ├── tz.py
        │   ├── tzwin.py
        │   └── zoneinfo
        │       ├── __init__.py
        │       └── zoneinfo-2010g.tar.gz
        ├── httplib2
        │   ├── cacerts.txt
        │   ├── __init__.py
        │   ├── iri2uri.py
        │   └── socks.py
        ├── __init__.py
        ├── oauth2client
        │   ├── anyjson.py
        │   ├── appengine.py
        │   ├── client.py
        │   ├── clientsecrets.py
        │   ├── crypt.py
        │   ├── django_orm.py
        │   ├── file.py
        │   ├── gce.py
        │   ├── __init__.py
        │   ├── keyring_storage.py
        │   ├── locked_file.py
        │   ├── multistore_file.py
        │   ├── old_run.py
        │   ├── tools.py
        │   ├── util.py
        │   └── xsrfutil.py
        └── uritemplate
            └── __init__.py

40 directories, 163 files
[root@centos-2gb-nyc3-01 deploy]#

[jmuharsky]

Hrm... apparently this was chaff from previous deployment work. I'm not sure why you're seeing that issue -- there clearly aren't anywhere near 10k files.

[michaelsmit]

Reviewing the detailed log for gcloud app deploy (for me, /home/$USER/.config/gcloud/logs/), it looks like it is trying to upload every file in PerfKitExplorer. Not sure why and didn't try to debug.

appcfg.py wasn't on my $PATH either, but I was able to deploy by finding the file and running it using its full path:
/home/$USER/google-cloud-sdk/platform/google_appengine/appcfg.py

Looks like the deployment process has changed since the README was written...