Loading
I have restored Gitlab from a backup, now every time I try to access the runners
webpage, I get a Internal Server error.
I have tried uninstalling all configured gitlab-runners and accessing it, the problem persists. Here is a trace from the gitlab-rails/production.log:
Started GET "/admin/runners" for 127.0.0.1 at 2019-01-16 07:17:12 -0500
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 66ms (ActiveRecord: 5.7ms)
ActionView::Template::Error ():
37:
38: .col-sm-6
39: .bs-callout
40: = render partial: 'ci/runner/how_to_setup_runner',
41: locals: { registration_token: Gitlab::CurrentSettings.runners_registration_token,
42: type: 'shared',
43: reset_token_url: reset_registration_token_admin_application_settings_path }
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:33:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field'
app/models/application_setting.rb:409:in `runners_registration_token'
lib/gitlab/current_settings.rb:19:in `method_missing'
app/views/admin/runners/index.html.haml:40:in `_app_views_admin_runners_index_html_haml___977288809002382166_69944849285200'
app/controllers/application_controller.rb:116:in `render'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:420:in `set_locale'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:19:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Started GET "/-/metrics" for 127.0.0.1 at 2019-01-16 07:17:24 -0500
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms)
asked Jan 16, 2019 at 12:18
1
I could resolve the issue by following official gitlab documentation:
gitlab-rails dbconsole
-- Clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- Clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- Clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;
-- Clear runner tokens
UPDATE ci_runners SET token = null, token_encrypted = null;
https://docs.gitlab.com/ee/raketasks/backup_restore.html#reset-runner-registration-tokens
Edit:
After clearing existing pipeline jobs (see above), I could still not open the ci settings page for some migrated projects where I had set environment variables. In this case try to remove them:
gitlab-rails dbconsole
SELECT * FROM ci_variables;
DELETE FROM ci_variables WHERE project_id='XX';
Hope that helps.
answered Oct 2, 2019 at 18:41
Tobias ErnstTobias Ernst
4,2141 gold badge32 silver badges30 bronze badges
1
You did a restore, but probably did not restore the secrets store (/etc/gitlab/gitlab-secrets.json or $GITLAB_HOME/config/secrets.json). Same issue as gitlab 500 errors in the admin area
You can reset it like this from the rails console on the gitlab server:
root@gitlab:/# gitlab-rails console
-------------------------------------------------------------------------------------
GitLab: 11.5.1 (c90ae59)
GitLab Shell: 8.4.1
postgresql: 9.6.8
-------------------------------------------------------------------------------------
Both Deployment and its :status machine have defined a different default for "status". Use only one or the other for defining defaults to avoid unexpected behaviors.
Loading production environment (Rails 4.2.10)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
=> true
answered Jan 16, 2019 at 19:55
5
i’ve solved this,maybe it is usefal
a. Reset CI/CD variables
gitlab-rails dbconsole
SELECT * FROM public."ci_group_variables";
SELECT * FROM public."ci_variables";
DELETE FROM ci_group_variables;
DELETE FROM ci_variables;
b. Reset runner registration tokens
gitlab-rails dbconsole
-- Clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- Clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- Clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
-- Clear runner tokens
UPDATE ci_runners SET token = null, token_encrypted = null;
c. Reset pending pipeline jobs
sudo gitlab-rails dbconsole
-- Clear build tokens
UPDATE ci_builds SET token = null, token_encrypted = null;
d. Fix project integrations
gitlab-rails dbconsole
-- truncate web_hooks table
TRUNCATE web_hooks CASCADE;
from gitlab
answered Mar 25, 2021 at 8:15
Here is what happen, i got either a 500 or 502 error after an upgrade using Omnibus method. And got the error message «Gitlab is not responding»
The first thing i did was to look for what could have happen. And the first place to see that is on the log file located at
/var/log/gitlab/gitlab-rails/production.log
And the log basically just give me an error as shown below,
Started GET "/users/sign_in" for 175.144.6.68 at 2015-12-12 23:48:54 +0800 Processing by SessionsController#new as HTML Completed 500 Internal Server Error in 98ms (ActiveRecord: 10.8ms) NoMethodError (undefined method `push_events=' for #<GitlabCiService:0x0000000463dba8>): app/models/project.rb:809:in `builds_enabled=' app/controllers/application_controller.rb:194:in `add_gon_variables'
But when i do a status check, it gives me this
[root@git gitlab-rails]# gitlab-ctl status run: gitlab-workhorse: (pid 4934) 1009s; run: log: (pid 4147) 1227s run: logrotate: (pid 4942) 1008s; run: log: (pid 296) 3434s run: nginx: (pid 4948) 1008s; run: log: (pid 299) 3434s run: postgresql: (pid 4957) 1007s; run: log: (pid 301) 3434s run: redis: (pid 4965) 1007s; run: log: (pid 294) 3434s run: sidekiq: (pid 4972) 1005s; run: log: (pid 302) 3434s run: unicorn: (pid 4990) 1004s; run: log: (pid 305) 3434s
I have pretty much no idea what is going on. But after trying out different ways, it seems to boil down to the following,
1. Check what is going on
Firing the following command should give you an idea what is going on with your configure.
sudo gitlab-rake gitlab:check
After that you could try see what is causing it.
2. Forget to turn on postgres before upgrade
Well, because gitlab said to shutdown gitlab before upgrading, hence i did this,
gitlab-ctl stop
which stops everything including postgres. Hence, database migration wasn’t possible. Therefore, i fire the following command and see whether that helps
gitlab-rake db:migrate
Now, after this i still got a 502 error but at least i’m not stuck with 500 error!
3. Forget to reconfigure after an upgrade
Well, if its not database migration, then every time you did a migration, remember to do a reconfigure!
gitlab-ctl reconfigure
Once i did this. Wait a while, and puff! The screen is back up!
I’m just grateful everything is ok! Just remember to back up your VM image before doing all these upgrades!
How to reproduce:
- login as admin
- go into the admin menu
- click on overview — runners
- or click save in the settings
Result: 500 Error: Whoops, something went wrong on our end.
Which system is running?
RHEL 7, outbound gitlab with Nginx and passengers
Errors:
OpenSSL::Cipher::CipherError ():
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:42:in `ensure_token'
ActionView::Template::Error ():
37:
38: .col-sm-6
39: .bs-callout
40: = render partial: 'ci/runner/how_to_setup_runner',
41: locals: { registration_token: Gitlab::CurrentSettings.runners_registration_token,
42: type: 'shared',
43: reset_token_url: reset_registration_token_admin_application_settings_path }
jww
97.9k91 gold badges413 silver badges887 bronze badges
asked Jan 10, 2019 at 11:47
3
I am almost sure it’s a wrong decryption key that gitlab reads from /etc/gitlab/gitlab-secrets.json (omnibus install) or $GITLAB_HOME/config/secrets.yml(source based install). If you don’t care about your runner_registration_token you can reset it like this from the console:
root@gitlab:/# gitlab-rails console
-------------------------------------------------------------------------------------
GitLab: 11.5.1 (c90ae59)
GitLab Shell: 8.4.1
postgresql: 9.6.8
-------------------------------------------------------------------------------------
Both Deployment and its :status machine have defined a different default for "status". Use only one or the other for defining defaults to avoid unexpected behaviors.
Loading production environment (Rails 4.2.10)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
=> true
irb(main):002:0>
answered Jan 10, 2019 at 20:10
1
In the GitLab 13.0 version, it has been confirmed that the command ApplicationSetting.current.reset_runners_registration_token!
is no longer supported. Previously, this command could be used to reset the registration token of GitLab CI/CD runners, but this functionality has been removed in later versions. Instead, GitLab administrators can reset the registration token of runners in the administration area of GitLab.
In GitLab versions 13.0 and later, if you attempt to use the command ApplicationSetting.current.reset_runners_registration_token!
, an error message will be displayed. This is because this command is no longer supported.
Source: GitLab 13.0 release notes
To resolve this issue in GitLab versions 13.0 and later, you can follow the steps below:
gitlab-rails console
> ApplicationSetting.first.delete
> ApplicationSetting.first
=> nil
Source: GitLab issue #57038
answered Jul 19 at 4:53
Problem Description:
My user wants to merge a request on our hosted GitLab EE 11.5.2 website.
But the website returns 500 Internal Server Error as follows:
The related log is as follows:
Completed 500 Internal Server Error in 56565ms (ActiveRecord: 746.1ms | Elasticsearch: 0.0ms)
ActionView::Template::Error (4:Deadline Exceeded):
42: %li.diffs-tab
44: Changes
45: %span.badge.badge-pill= @merge_request.diff_size
46:
47: #diff-notes-app.tab-content
48: #new.commits.tab-pane.active
lib/gitlab/gitaly_client/diff_stitcher.rb:13:in `each'
lib/gitlab/gitaly_client/diff_stitcher.rb:13:in `each'
lib/gitlab/git/diff_collection.rb:107:in `each_gitaly_patch'
lib/gitlab/git/diff_collection.rb:47:in `each'
lib/gitlab/git/diff_collection.rb:83:in `each_with_index'
lib/gitlab/git/diff_collection.rb:83:in `decorate!'
lib/gitlab/diff/file_collection/base.rb:32:in `diff_files'
lib/gitlab/diff/file_collection/base.rb:9:in `real_size'
app/models/merge_request.rb:411:in `diff_size'
app/views/projects/merge_requests/creations/_new_submit.html.haml:45:in `block in _app_views_projects_merge_requests_creations__new_submit_html_haml__4084413277917201848_70167460469980'
app/views/projects/merge_requests/creations/_new_submit.html.haml:43:in `_app_views_projects_merge_requests_creations__new_submit_html_haml__4084413277917201848_70167460469980'
app/views/projects/merge_requests/creations/new.html.haml:6:in `_app_views_projects_merge_requests_creations_new_html_haml__4204914316377908944_70167458972060'
app/controllers/application_controller.rb:116:in `render'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:427:in `set_locale'
lib/gitlab/middleware/multipart.rb:101:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
ee/lib/gitlab/jira/middleware.rb:15:in `call'
lib/gitlab/middleware/go.rb:17:in `call'
lib/gitlab/etag_caching/middleware.rb:11:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:22:in `call'
lib/gitlab/metrics/rack_middleware.rb:15:in `block in call'
lib/gitlab/metrics/transaction.rb:53:in `run'
lib/gitlab/metrics/rack_middleware.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:16:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
lib/gitlab/middleware/release_env.rb:10:in
Could you please tell me how to troubleshoot this problem?
Thank you.
Solution – 1
Eventually, we found the solution!
Though the merge request can be achieved by git command, it’s not recommended by our security policy.
And the solution is that the merge request should be operated via «Merge Requests» instead of «Repository > Branches«.
The successful way:
The failed way: