|
Replies:
16
-
Last Post:
Feb 28, 2009 6:49 AM
by: sparcdr
|
|
|
Posts:
12
From:
Registered:
11/20/06
|
|
|
|
how to create a fully functional mirror of pkg.opensolaris.org
Posted:
Feb 6, 2009 1:02 PM
To: Projects » indiana » discuss
Cc: Projects » caiman » discuss
|
|
hello
as I've been trying to get a working IPS mirror including the metadata for quite some time, I thought I would post it how to do it if someone needs to do this as well
mkdir pkg
cd pkg
pkgrecv -s http://pkg.opensolaris.org -n > opensol.index
for i in `cat opensol.index`; do pkgrecv -s http://pkg.opensolaris.org $i; done
for i in `cat opensol.index`
do
pkg=`echo $i|awk -F/ '{print $2}' |awk -F@ '{print $1}'`
echo import $pkg $i
trans=`pkgsend -s http://localhost:10000 open $i`
$trans
pkgsend -s http://localhost:10000 include -d $pkg $pkg/manifest
pkgsend -s http://localhost:10000 close
done
credits go to Karthik S S for the fetching part
regards
vinz
|
|
|
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of
pkg.opensolaris.org
Posted:
Feb 6, 2009 2:08 PM
in response to: vmeier
|
|
vinz meier wrote:
> hello
> as I've been trying to get a working IPS mirror including the metadata for quite some time, I thought I would post it how to do it if someone needs to do this as well
>
> mkdir pkg
> cd pkg
> pkgrecv -s http://pkg.opensolaris.org -n > opensol.index
> for i in `cat opensol.index`; do pkgrecv -s http://pkg.opensolaris.org $i; done
>
> for i in `cat opensol.index`
> do
> pkg=`echo $i|awk -F/ '{print $2}' |awk -F@ '{print $1}'`
> echo import $pkg $i
> trans=`pkgsend -s http://localhost:10000 open $i`
> $trans
> pkgsend -s http://localhost:10000 include -d $pkg $pkg/manifest
> pkgsend -s http://localhost:10000 close
> done
>
>
> credits go to Karthik S S for the fetching part
It is preferred that you not do this for a number of reasons:
1) the existing package servers are not provisioned for mass retrievals
in this fashion
2) this process will *not* give you a true *mirror* of the contents of
pkg.opensolaris.org since re-publishing the packages to a local
repository will give them a new timestamp
3) it's a large waste of bandwidth as you'll be retrieving every version
of every package (for example, you'll be getting a copy of each package
for every build; .86, .89, .97, .100, .101, etc.).
3) the officially supported method for mirroring can be found here:
http://opensolaris.org/os/project/pkg/Mirroring/
If you still choose to do this, I would ask that you do so at 0200 your
time in consideration of others.
Thank you,
--
Shawn Walker
_______________________________________________
indiana-discuss mailing list
indiana-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/indiana-discuss
|
|
|
|
|
Posts:
12
From:
Registered:
11/20/06
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of pkg.opensolaris.org
Posted:
Feb 7, 2009 4:17 PM
in response to: Shawn Walker
To: Projects » indiana » discuss
|
|
hello Shawn
i did not write the reason why i do need to do this:
as we need to install alot of workstations using AI each week for education purposes, it is alot more efficient to have a local mirror including the metadata.
installing a workstation with AI without a local mirror it takes 2-3 hours; installing a workstation with a local mirrror without the metadata it takes 40 minutes; installing a workstation with a local mirror including the metadata takes 20 minutes.
as the AI does not support adding a mirror (the syntax for the manifest exists, put it is not implemented), i was sort of forced to find another solution.
another benefit of having the local metadata is for the reason that if a new branch is published and the mirror is not updated (as the sync'ing takes 12 hours), the AI will fail as the mirror does not have the data and there is no failover to the main repository. and there is no option to specify the branch to be installed in AI (yet?)
but i agree, for most szenarios the official mirror solution is to be used
|
|
|
|
|
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of
pkg.opensolaris.org
Posted:
Feb 7, 2009 4:54 PM
in response to: vmeier
|
|
vinz meier wrote:
> hello Shawn
>
> i did not write the reason why i do need to do this:
> as we need to install alot of workstations using AI each week for education purposes, it is alot more efficient to have a local mirror including the metadata.
> installing a workstation with AI without a local mirror it takes 2-3 hours; installing a workstation with a local mirrror without the metadata it takes 40 minutes; installing a workstation with a local mirror including the metadata takes 20 minutes.
>
> as the AI does not support adding a mirror (the syntax for the manifest exists, put it is not implemented), i was sort of forced to find another solution.
>
> another benefit of having the local metadata is for the reason that if a new branch is published and the mirror is not updated (as the sync'ing takes 12 hours), the AI will fail as the mirror does not have the data and there is no failover to the main repository. and there is no option to specify the branch to be installed in AI (yet?)
>
> but i agree, for most szenarios the official mirror solution is to be used
I understand that there are many positive reasons for doing this, but
the method you are using has a fairly negative effect on packaging
server provision. I would strongly encourage you to wait for official
full mirroring support and stick with the content mirroring mechanisms
we already have in place.
Thanks for understanding,
--
Shawn Walker
_______________________________________________
indiana-discuss mailing list
indiana-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/indiana-discuss
|
|
|
|
|
Posts:
12
From:
Registered:
11/20/06
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of pkg.opensolaris.org
Posted:
Feb 8, 2009 3:37 AM
in response to: Shawn Walker
To: Projects » indiana » discuss
|
|
actually, i only needed to make the initial fetching, i will not be keeping the local mirror accurate; without the metadata i would be forced to be keeping the mirror accurate for the above reason (incapability of specifying a branch to be installed)
|
|
|
|
|
Posts:
1,708
From:
CZ
Registered:
2/5/07
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of
pkg.opensolaris.org
Posted:
Feb 9, 2009 1:49 AM
in response to: vmeier
|
|
Hi Vinz,
vinz meier wrote:
> hello Shawn
>
> i did not write the reason why i do need to do this:
> as we need to install alot of workstations using AI each week for education purposes, it is alot more efficient to have a local mirror including the metadata.
> installing a workstation with AI without a local mirror it takes 2-3 hours; installing a workstation with a local mirrror without the metadata it takes 40 minutes; installing a workstation with a local mirror including the metadata takes 20 minutes.
>
> as the AI does not support adding a mirror (the syntax for the manifest exists, put it is not implemented),
just to let you know, the bug tracking this problem was recently filed :-)
6435 AI should support installation from IPS content mirror
> i was sort of forced to find another solution.
>
> another benefit of having the local metadata is for the reason that if a new branch is published and the mirror is not updated (as the sync'ing takes 12 hours), the AI will fail as the mirror does not have the data and there is no failover to the main repository.
I think it shouldn't fail. If I understand correctly
how content mirrors work, if data is not found in
content mirror, they are pulled from main repo.
> and there is no option to specify the branch to be installed in AI (yet?)
You are right that dedicated AI manifest tag which would hold
branch to be installed is currently not implemented - please feel
free to file an enhancement in development/installer/autoinstall
category to capture this requirement.
That said, I haven't tried, but I think for now you could accomplish
this by explicitly list packages to be installed with branch
appended. Something like:
...
<ai_packages>
<package_name>
SUNWcsd@0.5.11-0.101
SUNWcs@0.5.11-0.101
slim_install@0.5.11-0.101
entire@0.5.11-0.101
</package_name>
</ai_packages>
...
Cheers,
Jan
_______________________________________________
indiana-discuss mailing list
indiana-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/indiana-discuss
|
|
|
|
|
Posts:
46
From:
Registered:
1/26/09
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of pkg.opensolaris.org
Posted:
Feb 10, 2009 4:52 PM
in response to: dambi
To: Projects » indiana » discuss
|
|
I was trying do do this as well, just as a one off copy, as a local cache makes the long download process while testing custom dvd builds faster and would reduce the load on the mirror site.
I saw the the mirroring info, but that was written for development.
What is the rsync command to mirror the release packages ?
|
|
|
|
|
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of
pkg.opensolaris.org
Posted:
Feb 11, 2009 9:02 AM
in response to: mark0001
|
|
Mark Bennett wrote:
> I was trying do do this as well, just as a one off copy, as a local cache makes the long download process while testing custom dvd builds faster and would reduce the load on the mirror site.
>
> I saw the the mirroring info, but that was written for development.
>
> What is the rsync command to mirror the release packages ?
Mirroring instructions can be found here:
http://opensolaris.org/os/project/pkg/Mirroring/
--
Shawn Walker
_______________________________________________
indiana-discuss mailing list
indiana-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/indiana-discuss
|
|
|
|
|
Posts:
46
From:
Registered:
1/26/09
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of pkg.opensolaris.org
Posted:
Feb 11, 2009 11:38 AM
in response to: Shawn Walker
To: Projects » indiana » discuss
|
|
Isn't this just the dev packages ?
or is the 2008.11 release included.
"The following instructions give a basic procedure for establishing a content mirror of http://pkg.opensolaris.org/dev, which serves the current development release of the OpenSolaris distribution."
I really only need a single branch eg. 2008.11. Is it possible to narrow the selection ?
|
|
|
|
|
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of
pkg.opensolaris.org
Posted:
Feb 11, 2009 12:26 PM
in response to: mark0001
|
|
Mark Bennett wrote:
> Isn't this just the dev packages ?
> or is the 2008.11 release included.
>
> "The following instructions give a basic procedure for establishing a content mirror of http://pkg.opensolaris.org/dev, which serves the current development release of the OpenSolaris distribution."
>
> I really only need a single branch eg. 2008.11. Is it possible to narrow the selection ?
Not at this time. I cannot tell you when that functionality will be
available.
Cheers,
--
Shawn Walker
_______________________________________________
indiana-discuss mailing list
indiana-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/indiana-discuss
|
|
|
|
|
Posts:
308
From:
US
Registered:
3/9/05
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror
of pkg.opensolaris.org
Posted:
Feb 12, 2009 9:25 AM
in response to: mark0001
|
|
* Mark Bennett <mark dot bennett at public dot co dot nz> [2009-02-11 19:39]:
> Isn't this just the dev packages ?
> or is the 2008.11 release included.
>
> "The following instructions give a basic procedure for establishing a content mirror of http://pkg.opensolaris.org/dev, which serves the current development release of the OpenSolaris distribution."
>
> I really only need a single branch eg. 2008.11. Is it possible to narrow the selection ?
release/ content is presently a subset of the dev/ content, so you can
use a dev/ mirror as a release/ mirror. We'll look at getting
separate access to the release/ content, though.
- Stephen
--
sch at sun dot com http://blogs.sun.com/sch/
_______________________________________________
indiana-discuss mailing list
indiana-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/indiana-discuss
|
|
|
|
|
Posts:
46
From:
Registered:
1/26/09
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of pkg.opensolaris.org
Posted:
Feb 12, 2009 7:46 PM
in response to: sch
To: Projects » indiana » discuss
|
|
That would be useful for both me and the repository.
Look forward to this being available.
Thanks guys.
|
|
|
|
|
Posts:
11
From:
Registered:
6/30/08
|
|
|
|
Re: how to create a fully functional mirror of pkg.opensolaris.org
Posted:
Feb 22, 2009 11:51 PM
in response to: vmeier
To: Projects » indiana » discuss
|
|
The rsync method introduced in the formal method "/usr/bin/rsync -a pkg.opensolaris.org::osol_dev_files /export/pkg/file
" does not work. In my side, there is no respond from the pkg.opensolaris.org server.
|
|
|
|
|
Posts:
308
From:
US
Registered:
3/9/05
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror
of pkg.opensolaris.org
Posted:
Feb 23, 2009 1:17 PM
in response to: autumnw
|
|
* SHOUJIN WANG <autumn dot wang at webex dot com> [2009-02-23 07:52]:
> The rsync method introduced in the formal method "/usr/bin/rsync -a
> pkg.opensolaris.org::osol_dev_files /export/pkg/file " does not work.
> In my side, there is no respond from the pkg.opensolaris.org server.
Use "rsync pkg.opensolaris.org::" to test basic responsiveness. A
listing of the available "modules" should be displayed, like the
following:
$ rsync pkg.opensolaris.org::
osol_dev_files "OpenSolaris Development Release Package Contents"
osol_release_isos "OpenSolaris Release ISO Images"
It's possible that there was an outage, or that too many rsync
sessions were open simultaneously. Without an explicit error message,
I can't diagnose further.
Thanks
Stephen
--
sch at sun dot com http://blogs.sun.com/sch/
_______________________________________________
indiana-discuss mailing list
indiana-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/indiana-discuss
|
|
|
|
|
Posts:
1,032
From:
US
Registered:
3/23/06
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of
pkg.opensolaris.org
Posted:
Feb 23, 2009 2:37 PM
in response to: sch
|
|
Stephen Hahn wrote:
> * SHOUJIN WANG <autumn dot wang at webex dot com> [2009-02-23 07:52]:
>
>> The rsync method introduced in the formal method "/usr/bin/rsync -a
>> pkg.opensolaris.org::osol_dev_files /export/pkg/file " does not work.
>> In my side, there is no respond from the pkg.opensolaris.org server.
>>
>
> Use "rsync pkg.opensolaris.org::" to test basic responsiveness. A
> listing of the available "modules" should be displayed, like the
> following:
>
> $ rsync pkg.opensolaris.org::
> osol_dev_files "OpenSolaris Development Release Package Contents"
> osol_release_isos "OpenSolaris Release ISO Images"
>
> It's possible that there was an outage, or that too many rsync
> sessions were open simultaneously. Without an explicit error message,
> I can't diagnose further.
>
> Thanks
> Stephen
>
>
I got some kind of python traceback when trying to update to b107... I
think the servers overloaded or something.
James
_______________________________________________
indiana-discuss mailing list
indiana-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/indiana-discuss
|
|
|
|
|
Posts:
308
From:
US
Registered:
3/9/05
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror
of pkg.opensolaris.org
Posted:
Feb 23, 2009 3:20 PM
in response to: sparcdr
|
|
* James Cornell <sparcdr at gmail dot com> [2009-02-23 22:38]:
> Stephen Hahn wrote:
> > * SHOUJIN WANG <autumn dot wang at webex dot com> [2009-02-23 07:52]:
> >
> >> The rsync method introduced in the formal method "/usr/bin/rsync -a
> >> pkg.opensolaris.org::osol_dev_files /export/pkg/file " does not work.
> >> In my side, there is no respond from the pkg.opensolaris.org server.
> >>
> >
> > Use "rsync pkg.opensolaris.org::" to test basic responsiveness. A
> > listing of the available "modules" should be displayed, like the
> > following:
> >
> > $ rsync pkg.opensolaris.org::
> > osol_dev_files "OpenSolaris Development Release Package Contents"
> > osol_release_isos "OpenSolaris Release ISO Images"
> >
> > It's possible that there was an outage, or that too many rsync
> > sessions were open simultaneously. Without an explicit error message,
> > I can't diagnose further.
>
> I got some kind of python traceback when trying to update to b107... I
> think the servers overloaded or something.
I suspect you hit a separate issue but, if this failure is
reproducible, please feel free to send me the output of the traceback.
(I don't have monitoring data that corroborates any outage/overload,
but I suppose it depends on when Shoujin and encountered your
respective problems.)
Cheers
Stephen
--
sch at sun dot com http://blogs.sun.com/sch/
_______________________________________________
indiana-discuss mailing list
indiana-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/indiana-discuss
|
|
|
|
|
Posts:
1,032
From:
US
Registered:
3/23/06
|
|
|
|
Re: [indiana-discuss] how to create a fully functional mirror of
pkg.opensolaris.org
Posted:
Feb 28, 2009 6:49 AM
in response to: sch
|
|
I hit a pkg bug, which is fixed by doing pfexec pkg install SUNWipkg before trying to do image-update after switching to B107+. I'm currently upgrading to B108. - James On Mon, Feb 23, 2009 at 5:20 PM, Stephen Hahn <sch at sun dot com> wrote:
I suspect you hit a separate issue but, if this failure is
reproducible, please feel free to send me the output of the traceback.
(I don't have monitoring data that corroborates any outage/overload,
but I suppose it depends on when Shoujin and encountered your
respective problems.)
Cheers
|
|
|
|
|
