Shell: xargs with a pipe to send logs to logstash

I have been playing with Logstash this weekend and I believe I have everything setup the way I want, so to put Kibana 3 to que test, I wanted to inject all my MediaTemple logs into Logstash to start searching and get used to querying Kibana.

Hence something like this was used:

 

find . -name '*.processed' -print0 | xargs -I {} -0 -i sh -c 'cat {} | netcat 127.0.0.1 50000'

 

This will find all my logs ending in ".processed" send them to xargs that will cat each file and pipe it to netcat to my logstash instance listening on port 50000.

 

I'l post my conf files here, but there's nothing to fancy and their documentation is fantastic if I take too long to post.

 

 

Hosting: Enabling Redis class on MediaTemple’s gridserver (gs)

Here's how I enabled access to a Redis server from my (GS) gridserver:

1. Store your SITEID in a variable:

export SITEID=`pwd | awk -F\/ '{ print $3 }'`

 

2. Create a folder to store the redis.so module

mkdir /home/$SITEID/data/lib
mkdir /home/$SITEID/data/lib/php/

 

3. Get the Redis PECL package (2.2.4 is the latest as of now)

wget http://pecl.php.net/get/redis-2.2.4.tgz

 

4. Uncompress it

tar -xvzf redis-2.2.4.tgz
cd redis-2.2.4

 

5. Compile it. I've set my (gs) to use PHP 5.5.5; change your accordingly

./configure --with-php-config=/usr/local/php-5.5.5/bin/php-config
make

 

6. "install" the module

cp modules/*.so /home/$SITEID/data/lib/php

 

7. Add the following to your /home/SITEID/etc/php.ini

;PECL
extension_dir=/home/97042/data/lib/php/
;redisphp
extension=redis.so

 

Check it:

Visit http://SITEID.gridserver.com/gs-bin/phpinfo.php-latest

 

redis-2-2-4-module

References

  • https://kb.mediatemple.net/questions/1682/Installing+PECL+extensions#gs
  • https://github.com/nicolasff/phpredis/issues/42
  • http://pecl.php.net/package/redis

Hosting: Backing up your MediaTemple using rsync and a linux box

Follow these steps to setup a decent backup solution for your MT hosting (you can adapt this to any other hosting that offers SSH and RSYNC access).

First you'll want your linux box to be able to SSH into your hosting without a password using a SSH key instead.

Then, just clone the following repository somewhere in your linux box

https://github.com/PortNumber53/mt-utils

Customize the file:

mt.sh.sample

Next, make sure the variables in

0-variables.sh

match your enviroment.

Then you just need to run:

mt.sh.sample

For extra peace of mind, set it to run from a cron job.

 

 

(gs)MT- REQUEST ID #680057 / STATUS: Pending Resolution

James S.

Thank you for the phone call. Here is a brief summary of our conversation:

You called because you were unable to access phpMyAdmin and phpPgAdmin from your AccountCenter and could not connect to the databases through a SSH client.

While looking into this issue I found that the migration from the MySQL Grid-Container back into the MySQL SmartPool failed due to a failed SSH connection during the process.  I was able to contact a systems administrator who started the migration process again, as we spoke we were able to verify that the migration had been completed and we had access to both phpMyAdmin and phpPgAdmin.

You mentioned that you thought this script should be updated so that if it fails such as it did earlier today it would automatically be run again.  A note of this has been made and sent to the (mt) Media Temple requests department for further review.

During the course of this call we also reviewed some issues with support requests regarding Account: 144100.  At this time I transferred you to speak with my lead technician to get more information regarding the status of the pg_dump error.

If you have any further questions regarding your (mt) Media Temple services, please feel free to contact us at any time.

Best Regards,

James Starbuck
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007
Check out our New User Forums!
http://kb.mediatemple.net/questions/824/

@
2009.10.23 05:54 PM
I’m waiting on this one

(gs)MT- REQUEST ID #670588 / STATUS: Resolved

Rob C.

Thank you for the phone call. Here is a brief summary of our conversation:

You called in to report that your Postgres server was not working. Your web application was displaying error messages, you couldn’g connect via the command-line and phpPgAdmin was not working. I notified an engineer and they are taking a look at it. I’ll be calling you back this evening to let you know the status.

If you have any further questions regarding your (mt) Media Temple services, please feel free to contact us at any time.

Best Regards,

Rob C.
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

NEW! User Forums: http://kb.mediatemple.net/questions/824/

@
2009.10.08 09:51 PM
Maurício

I have posted a screen capture here:
http://portnumber53.com/images/alocate_memory.jpg

waiting for your call back

@
2009.10.08 09:54 PM
Rob C.

Thank you for the phone call. Here is a brief summary of our conversation:

You had some concerns you wanted addressed during our call:

1) Regarding support request #657283 you’d like to know if the problem with pg_dump versions has been fixed because you want to move back to the Smart Pool and are concerned about database corruption.

2) Your Container that has both MySQL and Postgres running ran into problems tonight and had to be rebooted. You’ve had the same application error displayed in the Smart Pool, after you were bursted and in a Container. I’m going to see if we have any Postgres server logs available.

3) You’d like someone to call you back about these issues.

If you have any further questions regarding your (mt) Media Temple services, please feel free to contact us at any time.

Best Regards,

Rob C.
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

NEW! User Forums: http://kb.mediatemple.net/questions/824/

@
2009.10.08 11:07 PM
Maurício

just a small detail, I had the same issues regardless the application I was using (psql, pg_dump, phpPgAdmin) in both the Smart Pool and the Grid Container.
None of my "Postgresql websites" have big databases (or tables) and they don’t have high traffic usage at the moment.

@
2009.10.08 11:25 PM
Mike M.

The application is giving errors about the environment being out of memory. This message can happen in the Smartpool or the Container because you are exhausting all the resources avaliable. The Postgres error logs have been put in the /backup directory and this information can be viewed via the Account Center File Manager. I would advise to look over this information in regards to optimizing the environment. If you need any further insight please to not hesitate to ask.

Best Regards,

Mike M
Customer Support
Check out the New User Forums: http://kb.mediatemple.net/questions/824/
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

@
2009.10.09 05:41 AM

(gs)MT- REQUEST ID #657010 / STATUS: Resolved

Robert V.

Thank you for the phone call. Here is a brief summary of our conversation:

Due to the repeated inconvenience of your PostgreSQL database running out of memory intermittently since you started this account, I am reassigning this Support Request to our Billing Department to add a database container to your account.  The first month will be free.  If you decide that you do not want to keep the container you will need to close the container before the month is over.  If you do not close the container before the month is over your account will be assessed a fee for the container’s usage.

Thank you for using (mt) Media Temple!  Feel free to let us know if there is anything else we can help you with.

Best Regards,

Robert V.
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

NEW! User Forums: http://kb.mediatemple.net/questions/824
Knowledge base: http://kb.mediatemple.net/

@
2009.09.17 01:40 PM
Shawnte A.

We have added a MySQL Container Lite to your account.  The billing cycle for this service is 10/25/09 at which time you will be charged $20.00.  If you have any additional questions or concerns, please feel free to contact us.

Best Regards,

Shawnte` A.
Billing Administrator
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

@
2009.09.17 03:38 PM
James S.

Thank you for the phone call. Here is a brief summary of our conversation:

You called because your databases were moved into a MySQL GridContainer,and since this has occurred your sites running in a PostgreSQL database are unable to resolve to the database content.  An error occurred when the databases were moved to the container which caused the tables in your PostgreSQL databases not to be added to the databases when they were created.

In order to have this issue resolved I have reassigned this support request to a higher level employee to resolve.  Please anticipate a delay while this support request is reassigned and resolved.

If you have any further questions regarding your (mt) Media Temple services, please feel free to contact us at any time.

Best Regards,

James Starbuck
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007
Check out our New User Forums!
http://kb.mediatemple.net/questions/824/

@
2009.09.17 05:32 PM
Maurício

In case I choose not to keep the Container, will my databases be moved automatically to the regular pool?

@
2009.09.17 05:33 PM
Michael H.

You were having a problem with your databases not being populated with tables after the GridContainer move. Our admins were able to recover your data and I restored the tables for you. We don’t currently have an answer for the original issue, but our engineers are working on it. You should receive an email once they have it resolved.

If you have any further questions regarding your (mt) Media Temple services, please feel free to contact us at any time.

Best Regards,

Mike H.
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

NEW! User Forums: http://kb.mediatemple.net/questions/824/

@
2009.09.17 08:51 PM
Mike M.

Please refer to your Grid-Service "aliofthevalley.com", Support Request # 657007, for the response to this Support Request. If you have any further questions regarding your (mt) Media Temple services, please feel free to contact us at any time.

Best Regards,

Mike M
Customer Support
Check out the New User Forums: http://kb.mediatemple.net/questions/824/
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

@
2009.09.17 11:17 PM
Travis O.

Thank you for the phone call. Here is a brief summary of our conversation:

You called because you were unhappy about the way your PostgreSQL databases were bursted and you lost the complete functionality of your databases.

We have determined that your request requires the assistance of a higher level staff member or system administrator. Please anticipate a short delay while we re-assign your support request to a staff member who is better equipped to service your needs.

Due to the nature of this support request additional time may be needed to provide you with a response. Thank you.

Best Regards,

Travis Oberlander
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

@
2009.09.24 10:54 AM
Nathaniel M.

Thank you for the phone call. You called in because you wanted to speak with a supervisor about an issue related to your pg databases. I asked if you could submit a support request with the exact problem so I could escalate it if necessary and you asked to speak to a supervisor. I then transferred you to TJ.

Best Regards,

Nat M
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007
User Forums: http://kb.mediatemple.net/questions/824/

@
2009.09.24 04:49 PM
Maurício

I understand I received a month credit for a Grid Container because of the allocate memory problems (which happen when trying to connect to the database server; and not white running any query either complex or simple)

I’d like to know if I could get any compensation on the fact that my databases were not moved correctly to the Grid Container.

@
2009.09.24 05:14 PM
Daniel C.

Thank you for the phone call. Here is a brief summary of our conversation:

We spoke today regarding issues connecting to your database externally. I was able to duplicate this issue and explained that an administrator will take a look into this shortly.

If you have any further questions regarding your (mt) Media Temple services, please feel free to contact us at any time.

Best Regards,

Daniel C.
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

NEW! User Forums: http://forums.mediatemple.net

@
2009.09.24 06:23 PM
Daniel C.

Please use support request #661555 for further information on the external connection issue.

Best Regards,

Daniel C.
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

NEW! User Forums: http://forums.mediatemple.net

@
2009.09.24 08:10 PM

(gs)MT- REQUEST ID #551946 / STATUS: Resolved

Maurício:

I have been receiving "Cannot allocate memory" errors when trying to connect to a postgresql database
I believe I’m not doing anything weird on my end, and since it’s a website I just started developing I’m probably the only one opening the page.

Can you guys look at what the logs say?

the database is db64704_dothejob
and the website is http://dothejobfor.me/store

thanks

@
2009.03.30 05:40 PM
Mike M.

We would have to see the active processes running in order to see the out of memory errors. Since the databases are shared there is no log we can go to to see your specific memory error. I was able to access ‘http://dothejobfor.me/store‘ without any problems. Is there anything specific I should be doing to replicate the memory error?

Best Regards,

Mike M
Customer Support
Check out the New User Forums: http://kb.mediatemple.net/questions/824/
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

@
2009.03.31 01:30 AM
Maurício
I just got the error message again:

pg_connect() [function.pg-connect]: Unable to connect to PostgreSQL server: could not fork new process for connection: Cannot allocate memory

started: 1:46am 9/14/2009

my mysql dies along:

There was an SQL error: Out of memory (Needed 122672 bytes) – SELECT *
FROM `music_song`
ORDER BY `songid` DESC
LIMIT 0, 10

and the problem lasted for 2 minutes

Sharon C.

These errors indicate that your scripts were trying to access the database server during the latency spike that occurred at about the time you documented. Due to the shared nature of the (gs) Grid-Service, there may be brief periods of resource unavailability. We have both automated and manual monitoring of the servers to keep these interruptions to a minimum; however, we apologize for the inconvenience. Basically, if several users are trying to use a higher-than-average amount of resources in the database SmartPool, we have automatic systems in place, and engineers who keep an eye on things, to move those high-load users to temporary isolated database servers, so that they can continue to function and other users won’t be affected. There can be a few minutes of resource unavailability before the load-balancing kicks in, however. If you want to read more about how MySQL on the (gs) Grid-Service works, please see this article:

http://blog.mediatemple.net/weblog/2007/01/19/anatomy-of-mysql-on-the-grid/

If this system does not appeal to you, we also have MySQL Containers (we do not have PostGreSQL Containers, however). A MySQL Container will do two things for you. First, it will isolate you from other users, so you won’t be affected by any "bad neighbors" in the SmartPool, even for brief periods of time. Second, it will allow you to run a more detailed analysis on all your queries, so you can be sure your own database use is fully optimized. It’s completely seamless to move to and from a Container. See these articles for more details:

Getting started with your MySQL GridContainer
http://kb.mediatemple.net/questions/705

Generating a MySQL report for your MySQL container
http://kb.mediatemple.net/questions/939

Best Regards,

Sharon C.
Customer Support
(mt) Media Temple
<v> 877-578-4000
<f> 310-564-2007

NEW! User Forums: http://kb.mediatemple.net/questions/824/

@
2009.09.14 11:00 PM

Storage problems when my account was on Cluster 02

Last year (2008), MediaTemple was having a lot of issues with their Cluster 02, which meant my websites (and everyone’s) on the cluster would be unavailable several times. After struggling for months they were able to narrow the problems to a defective segment of their storage components. By then I had spent hours on the phone with them, noticed that Cluster 01 would also have its issues; but I never saw any issues issues with Cluster 03.

It was then, that occurred to me to ask them to move my account to said cluster!

Unfortunatelly they couldn’t do it automagically, which I was ok with doing manually since they gave me an one month credit to do so. Great so far.

If I remember right, I managed to move everything working late hours spread over 2 nights. Since all the transfers were inside their network I’d get speed transfers liks 30MB/s which really saves a *lot* of time.

First *HUGE* problem.

Right after I moved and had everything working on the new (gs) on Cluster 03, I called them to expedite the old (gs)cancellation. That was when the technician clicked the wrong (gs) and deleted the new (gs) server… Need I say I yelled at him?

Well, he said he would escalate my ticket to the administrators to have them restore my server from their graveyard (he said they mantain the data somewhere else even after cancelling accounts, probably, for cases like this one). When I asked when that ticket would be processed, he game the standard "ASAP" response. And I said…. Wait! You deleted my account and now I have to wait??

Around 4 hours later (the administrator was having lunch and no one else was taking care of their hundreds of thousands of customers), I finally have my data back.

Now this part, I don’t remember very well, but if my brain is not playing tricks on me, they could not recover everything. And so I still had some tweaks and adjustments to have my websites up and running smoothly again….

At around 11 p.m. that day, I had everything working again…. so I called them to get rid of the now old (gs) server (the one they killed by accident). I asked whoever answered the phone that time to please read the screen very carefully so he wouldn’t kill the wrong (gs) server…. (yes, I tend to be sarcastic)

Whatever he did there…. my server disappeared one more time….  everything was down again….

At this point I turned into the kind of customer that can only ask: "Are you guys playing around there?"

So he has to check went happened and calls me back around 30 minutes later, and said something about simply deactivating the account (as opposed to killing it) and my websites were up and working again.

Customer support until that day had been really helpful and seemed like they knew what they were doing.

From this day, though, I kinda lost confidence and have always argued back.

BTW, I asked for some compensation for the first technician having deleted my (gs) and that almost happening a second time, no administrator present to solve my problem. I got 1 month of credit….. NOT A YEAR, SIMPLY 1 MONTH CREDIT.  That’s how important my websites and data are…. $20 for MediaTemple.

  • IGlr