[Check_mk (english)] Proxy Errors

[Dhawal Doshy]

Hello List,

We appear to getting a lot of these errors in WATO.

HTTP Error - 502:

502 Proxy Error

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
POST /poller02/check_mk/automation.py.
Reason: Error reading from remote server

Apache/2.2.15 (CentOS) Server at poller02.mydomain.com Port 443

This doesn't happen all the time, but quite frequently.

We've already implemented liveproxyd, but that doesn't help for multisite.

Running OMD 1.1 on CentOS 6.5 64-bit.

- dhawal

[Dhawal Doshy]

Additionally, sometimes though not always, the Proxy Error happens
along with a "Write failed, client closed connection.", Google
suggests this may be a mod_python error.

==> /omd/sites/master01/var/log/apache/error_log <==
[Thu Mar 06 20:03:56 2014] [warn] [client 127.0.0.1] Write failed,
client closed connection., referer:
http://master01.mydomain.com/master01/check_mk/wato.py?mode=changelog&_site=poller02&_siteaction=restart&_transid=1394116287/927618181&folder=

Post by Dhawal Doshy
Hello List,
We appear to getting a lot of these errors in WATO.
502 Proxy Error
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
POST&nbsp;/poller02/check_mk/automation.py.
Reason: Error reading from remote server
Apache/2.2.15 (CentOS) Server at poller02.mydomain.com Port 443
This doesn't happen all the time, but quite frequently.
We've already implemented liveproxyd, but that doesn't help for multisite.
Running OMD 1.1 on CentOS 6.5 64-bit.
- dhawal

[Jimw]

Post by Dhawal Doshy
Hello List,
We appear to getting a lot of these errors in WATO.
502 Proxy Error
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
POST /poller02/check_mk/automation.py.
Reason: Error reading from remote server
Apache/2.2.15 (CentOS) Server at poller02.mydomain.com Port 443
This doesn't happen all the time, but quite frequently.
We've already implemented liveproxyd, but that doesn't help for multisite.
Running OMD 1.1 on CentOS 6.5 64-bit.
- dhawal

We've had problems committing changes to WATO that have gotten worse over
time. I suspect it's taking longer and longer to compile the checks, verify
the config, and update the db as we add hosts and services. I increased the
settings in /etc/php.ini and it seems to help with the proxy timeouts.

;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;

; Maximum execution time of each script, in seconds
; http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time
;max_execution_time = 30
max_execution_time = 120

; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate
unexpectedly
; long running scripts.
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://www.php.net/manual/en/info.configuration.php#ini.max-input-time
;max_input_time = 60
max_input_time = 120

; Maximum input variable nesting level
;
http://www.php.net/manual/en/info.configuration.php#ini.max-input-nesting-level
;max_input_nesting_level = 64

; Maximum amount of memory a script may consume (128MB)
; http://www.php.net/manual/en/ini.core.php#ini.memory-limit
;memory_limit = 128M
memory_limit = 256M

[Bhavin Ajmera]

Hi Jimw,

I have changed the above settings in /etc/php.ini as well as in
/opt/omd/sites/sitename/etc/apache/php.ini but no luck still same error.
Just a query all the pages are python pages like wato.py, index.py,
dashboard.py, etc changing PHP setting would help to resolve this error ?
Thanks Jimw for your inputs.

Regards,
Bhavin Ajmera

Post by Dhawal Doshy

Post by Dhawal Doshy
Hello List,
We appear to getting a lot of these errors in WATO.
502 Proxy Error
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
POST /poller02/check_mk/automation.py.
Reason: Error reading from remote server
Apache/2.2.15 (CentOS) Server at poller02.mydomain.com Port 443
This doesn't happen all the time, but quite frequently.
We've already implemented liveproxyd, but that doesn't help for

multisite.

Post by Dhawal Doshy
Running OMD 1.1 on CentOS 6.5 64-bit.
- dhawal

We've had problems committing changes to WATO that have gotten worse over
time. I suspect it's taking longer and longer to compile the checks, verify
the config, and update the db as we add hosts and services. I increased the
settings in /etc/php.ini and it seems to help with the proxy timeouts.
;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;
; Maximum execution time of each script, in seconds
;
http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time
;max_execution_time = 30
max_execution_time = 120
; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate
unexpectedly
; long running scripts.
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://www.php.net/manual/en/info.configuration.php#ini.max-input-time
;max_input_time = 60
max_input_time = 120
; Maximum input variable nesting level
;
http://www.php.net/manual/en/info.configuration.php#ini.max-input-nesting-level
;max_input_nesting_level = 64
; Maximum amount of memory a script may consume (128MB)
; http://www.php.net/manual/en/ini.core.php#ini.memory-limit
;memory_limit = 128M
memory_limit = 256M
_______________________________________________
checkmk-en mailing list
http://lists.mathias-kettner.de/mailman/listinfo/checkmk-en

[Gary Herbstman]

I get a similar error from WATO when inventorying a large switch stack. It is a timeout issue. It I run the command line inventory it works fine.

From: checkmk-en-***@lists.mathias-kettner.de [mailto:checkmk-en-***@lists.mathias-kettner.de] On Behalf Of Bhavin Ajmera
Sent: Friday, May 23, 2014 04:44
To: checkmk-***@lists.mathias-kettner.de
Subject: Re: [Check_mk (english)] Proxy Errors

Hi Jimw,

I have changed the above settings in /etc/php.ini as well as in /opt/omd/sites/sitename/etc/apache/php.ini but no luck still same error. Just a query all the pages are python pages like wato.py, index.py, dashboard.py, etc changing PHP setting would help to resolve this error ? Thanks Jimw for your inputs.

Regards,
Bhavin Ajmera

Post by Dhawal Doshy
Hello List,
We appear to getting a lot of these errors in WATO.
502 Proxy Error
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
POST /poller02/check_mk/automation.py.
Reason: Error reading from remote server
Apache/2.2.15 (CentOS) Server at poller02.mydomain.com<http://poller02.mydomain.com> Port 443
This doesn't happen all the time, but quite frequently.
We've already implemented liveproxyd, but that doesn't help for multisite.
Running OMD 1.1 on CentOS 6.5 64-bit.
- dhawal

We've had problems committing changes to WATO that have gotten worse over
time. I suspect it's taking longer and longer to compile the checks, verify
the config, and update the db as we add hosts and services. I increased the
settings in /etc/php.ini and it seems to help with the proxy timeouts.

;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;

; Maximum execution time of each script, in seconds
; http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time
;max_execution_time = 30
max_execution_time = 120

; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate
unexpectedly
; long running scripts.
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://www.php.net/manual/en/info.configuration.php#ini.max-input-time
;max_input_time = 60
max_input_time = 120

; Maximum input variable nesting level
;
http://www.php.net/manual/en/info.configuration.php#ini.max-input-nesting-level
;max_input_nesting_level = 64

; Maximum amount of memory a script may consume (128MB)
; http://www.php.net/manual/en/ini.core.php#ini.memory-limit
;memory_limit = 128M
memory_limit = 256M





_______________________________________________
checkmk-en mailing list
checkmk-***@lists.mathias-kettner.de<mailto:checkmk-***@lists.mathias-kettner.de>
http://lists.mathias-kettner.de/mailman/listinfo/checkmk-en

[Bhavin Ajmera]

I have successfully done an inventory of 813 services on a single device
from wato without any issues, I face an issue only while activating changes
through wato I have 20 slave connecting to a single master.

Post by Gary Herbstman
I get a similar error from WATO when inventorying a large switch stack.
It is a timeout issue. It I run the command line inventory it works fine.
*Sent:* Friday, May 23, 2014 04:44
*Subject:* Re: [Check_mk (english)] Proxy Errors
Hi Jimw,
I have changed the above settings in /etc/php.ini as well as in
/opt/omd/sites/sitename/etc/apache/php.ini but no luck still same error.
Just a query all the pages are python pages like wato.py, index.py,
dashboard.py, etc changing PHP setting would help to resolve this error ?
Thanks Jimw for your inputs.
Regards,
Bhavin Ajmera

Post by Dhawal Doshy
Hello List,
We appear to getting a lot of these errors in WATO.
502 Proxy Error
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
POST /poller02/check_mk/automation.py.
Reason: Error reading from remote server
Apache/2.2.15 (CentOS) Server at poller02.mydomain.com Port 443
This doesn't happen all the time, but quite frequently.
We've already implemented liveproxyd, but that doesn't help for

multisite.

Post by Dhawal Doshy
Running OMD 1.1 on CentOS 6.5 64-bit.
- dhawal

We've had problems committing changes to WATO that have gotten worse over
time. I suspect it's taking longer and longer to compile the checks, verify
the config, and update the db as we add hosts and services. I increased the
settings in /etc/php.ini and it seems to help with the proxy timeouts.
;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;
; Maximum execution time of each script, in seconds
;
http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time
;max_execution_time = 30
max_execution_time = 120
; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate
unexpectedly
; long running scripts.
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://www.php.net/manual/en/info.configuration.php#ini.max-input-time
;max_input_time = 60
max_input_time = 120
; Maximum input variable nesting level
;
http://www.php.net/manual/en/info.configuration.php#ini.max-input-nesting-level
;max_input_nesting_level = 64
; Maximum amount of memory a script may consume (128MB)
; http://www.php.net/manual/en/ini.core.php#ini.memory-limit
;memory_limit = 128M
memory_limit = 256M
_______________________________________________
checkmk-en mailing list
http://lists.mathias-kettner.de/mailman/listinfo/checkmk-en

[Jim Welch]

Sorry, we have the same sort of problems with nagios xi, which uses php and
changing the php.ini settings worked for it. I tried the same settings on our
OMD server and it *seemed* to work (the errors are intermittent). I haven't
had time to try and find out exactly how check_mk's web interface works
so I don't know if it's an apache issue, python, or mysql.

I did see that the proxy timeout msg is controlled by
the setting in /etc/httpd/conf/httpd.conf (at least on my RHEL 6 server):

#
# Timeout: The number of seconds before receives and sends time out.
#
#Timeout 30
#Timeout 60
Timeout 120

Though I don't know what is going on under the hood to cause the requests
to take so long. Maybe there's a problem with my db and it needs to be
cleaned up...I do know that the web GUI frequently gets slow, drops input
characters, fails to recognize links, and requires multiple clicks on an icon
or link to activate. Something has it tied up, but I'm not sure what.
1 server, no slaves
~700 hosts ~16000 services
load avg usually between 5-10 (i know...we're looking at upgrading the hw)
not running out of memory

----- Original Message -----
From: Bhavin Ajmera <***@gmail.com>
To: checkmk-***@lists.mathias-kettner.de
Sent: Fri, 23 May 2014 04:43:46 -0400 (EDT)
Subject: Re: [Check_mk (english)] Proxy Errors


Hi Jimw,


I have changed the above settings in /etc/php.ini as well as in /opt/omd/sites/sitename/etc/apache/php.ini but no luck still same error. Just a query all the pages are python pages like wato.py, index.py, dashboard.py, etc changing PHP setting would help to resolve this error ? Thanks Jimw for your inputs.


Regards,
Bhavin Ajmera

Post by Dhawal Doshy
Hello List,
We appear to getting a lot of these errors in WATO.
502 Proxy Error
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
POST /poller02/check_mk/automation.py.
Reason: Error reading from remote server
Apache/2.2.15 (CentOS) Server at poller02.mydomain.com Port 443
This doesn't happen all the time, but quite frequently.
We've already implemented liveproxyd, but that doesn't help for multisite.
Running OMD 1.1 on CentOS 6.5 64-bit.
- dhawal

We've had problems committing changes to WATO that have gotten worse over

time. I suspect it's taking longer and longer to compile the checks, verify

the config, and update the db as we add hosts and services. I increased the

settings in /etc/php.ini and it seems to help with the proxy timeouts.


;;;;;;;;;;;;;;;;;;;

; Resource Limits ;

;;;;;;;;;;;;;;;;;;;


; Maximum execution time of each script, in seconds

; http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time

;max_execution_time = 30

max_execution_time = 120


; Maximum amount of time each script may spend parsing request data. It's a good

; idea to limit this time on productions servers in order to eliminate

unexpectedly

; long running scripts.

; Default Value: -1 (Unlimited)

; Development Value: 60 (60 seconds)

; Production Value: 60 (60 seconds)

; http://www.php.net/manual/en/info.configuration.php#ini.max-input-time

;max_input_time = 60

max_input_time = 120


; Maximum input variable nesting level

;
http://www.php.net/manual/en/info.configuration.php#ini.max-input-nesting-level

;max_input_nesting_level = 64


; Maximum amount of memory a script may consume (128MB)

; http://www.php.net/manual/en/ini.core.php#ini.memory-limit

;memory_limit = 128M

memory_limit = 256M






_______________________________________________

checkmk-en mailing list
checkmk-***@lists.mathias-kettner.de
http://lists.mathias-kettner.de/mailman/listinfo/checkmk-en

[Bhavin Ajmera]

Hi All,

I have managed to solve the 502 proxy error atleast on my setup. This
error is caused when check_mk which run on port 5000 using mod_proxy of
apache. When check_automation.py is ran using apache which passes the
request to mod_proxy on port 5000, check_mk timesout giving this error. I
have changes the following file which works great on my setup from couple
of days.
"/omd/sites/sitename/etc/apache/proxy-port.conf" I have added the following
text in bold italic.

<IfModule mod_proxy_http.c>
<Proxy http://127.0.0.1:5000/sitename>
Order allow,deny
allow from all
</Proxy>

*<IfModule mod_headers.c>*
*RequestHeader unset Expect early*
*</IfModule>*

<Location /poller12>
ProxyPass http://127.0.0.1:5000/sitename retry=1 disablereuse=On
*timeout=600
Keepalive=On*
ProxyPassReverse http://127.0.0.1:5000/sitename

* SetEnv force-proxy-request-1.0 1*
* SetEnv proxy-nokeepalive 1*
</Location>
</IfModule>

I hope this helps someone who has same issue like me.

Regards,
Bhavin Ajmera

Post by Jim Welch
Sorry, we have the same sort of problems with nagios xi, which uses php and
changing the php.ini settings worked for it. I tried the same settings on our
OMD server and it *seemed* to work (the errors are intermittent). I haven't
had time to try and find out exactly how check_mk's web interface works
so I don't know if it's an apache issue, python, or mysql.
I did see that the proxy timeout msg is controlled by
#
# Timeout: The number of seconds before receives and sends time out.
#
#Timeout 30
#Timeout 60
Timeout 120
Though I don't know what is going on under the hood to cause the requests
to take so long. Maybe there's a problem with my db and it needs to be
cleaned up...I do know that the web GUI frequently gets slow, drops input
characters, fails to recognize links, and requires multiple clicks on an icon
or link to activate. Something has it tied up, but I'm not sure what.
1 server, no slaves
~700 hosts ~16000 services
load avg usually between 5-10 (i know...we're looking at upgrading the hw)
not running out of memory
----- Original Message -----
Sent: Fri, 23 May 2014 04:43:46 -0400 (EDT)
Subject: Re: [Check_mk (english)] Proxy Errors
Hi Jimw,
I have changed the above settings in /etc/php.ini as well as in
/opt/omd/sites/sitename/etc/apache/php.ini but no luck still same error.
Just a query all the pages are python pages like wato.py, index.py,
dashboard.py, etc changing PHP setting would help to resolve this error ?
Thanks Jimw for your inputs.
Regards,
Bhavin Ajmera

Post by Dhawal Doshy

Post by Dhawal Doshy
Hello List,
We appear to getting a lot of these errors in WATO.
502 Proxy Error
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
POST /poller02/check_mk/automation.py.
Reason: Error reading from remote server
Apache/2.2.15 (CentOS) Server at poller02.mydomain.com Port 443
This doesn't happen all the time, but quite frequently.
We've already implemented liveproxyd, but that doesn't help for

multisite.

Post by Dhawal Doshy
Running OMD 1.1 on CentOS 6.5 64-bit.
- dhawal

We've had problems committing changes to WATO that have gotten worse over
time. I suspect it's taking longer and longer to compile the checks, verify
the config, and update the db as we add hosts and services. I increased the
settings in /etc/php.ini and it seems to help with the proxy timeouts.
;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;
; Maximum execution time of each script, in seconds
;
http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time
;max_execution_time = 30
max_execution_time = 120
; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate
unexpectedly
; long running scripts.
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://www.php.net/manual/en/info.configuration.php#ini.max-input-time
;max_input_time = 60
max_input_time = 120
; Maximum input variable nesting level
;
http://www.php.net/manual/en/info.configuration.php#ini.max-input-nesting-level
;max_input_nesting_level = 64
; Maximum amount of memory a script may consume (128MB)
; http://www.php.net/manual/en/ini.core.php#ini.memory-limit
;memory_limit = 128M
memory_limit = 256M
_______________________________________________
checkmk-en mailing list
http://lists.mathias-kettner.de/mailman/listinfo/checkmk-en